A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking

Lorenzo Papa; Paolo Russo; Irene Amerini; Luping Zhou

Source-linked AI summary

A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking

Lorenzo Papa, Paolo Russo, Irene Amerini, Luping Zhou

arXiv:2309.02031v2Sep 5, 2023cs.CV

TL;DR

ViTs offer global information through self-attention but become difficult to deploy as resolution, model size, and computational demands grow. This survey synthesizes four efficiency strategy categories, compares them across benchmarks with the EER metric, and identifies trade-offs alongside open challenges.

Problem
ViT self-attention has computational and memory costs that grow quadratically with image resolution, complicating deployment under real-world resource constraints.
Method
The survey mathematically formalizes and reviews compact architecture, pruning, knowledge distillation, and quantization strategies, then compares them using benchmark datasets and EER.
Results
Across reviewed methods, the survey compares accuracy–efficiency trade-offs and identifies preferred strategies, including NoisyQuant-PTQ4ViT-Swin-B as the most efficient analyzed quantized model at 84.7% Top-1 accuracy.
Takeaways & Limitations
EER provides a unified way to assess efficiency factors affecting inference devices alongside estimation performance across reviewed ViT strategies.
Takeaways & Limitations
The survey identifies limited resource-constrained-device benchmarks and insufficient study of combined efficiency strategies as open limitations.

Abstract

from arXiv · show

Vision Transformer (ViT) architectures are becoming increasingly popular and widely employed to tackle computer vision applications. Their main feature is the capacity to extract global information through the self-attention mechanism, outperforming earlier convolutional neural networks. However, ViT deployment and performance have grown steadily with their size, number of trainable parameters, and operations. Furthermore, self-attention's computational and memory cost quadratically increases with the image resolution. Generally speaking, it is challenging to employ these architectures in real-world applications due to many hardware and environmental restrictions, such as processing and computational capabilities. Therefore, this survey investigates the most efficient methodologies to ensure sub-optimal estimation performances. More in detail, four efficient categories will be analyzed: compact architecture, pruning, knowledge distillation, and quantization strategies. Moreover, a new metric called Efficient Error Rate has been introduced in order to normalize and compare models' features that affect hardware devices at inference time, such as the number of parameters, bits, FLOPs, and model size. Summarizing, this paper firstly mathematically defines the strategies used to make Vision Transformer efficient, describes and discusses state-of-the-art methodologies, and analyzes their performances over different application scenarios. Toward the end of this paper, we also discuss open challenges and promising research directions.

1 INTRODUCTION

Vision Transformers capture global image information through self-attention but face rapidly increasing computational and memory demands, especially at high resolutions. This survey reviews four efficiency strategies and introduces EER to compare their accuracy–efficiency trade-offs.

Motivation: Self-attention’s computational and memory cost increases quadratically with image resolution, creating a major deployment challenge for ViTs.The challenge is especially relevant to high-resolution and dense prediction tasks.
Motivation: Earlier surveys provided limited coverage of efficient ViT methodologies, motivating a dedicated and systematic review.The paper contrasts its scope with prior studies that focused on general transformers or limited efficient-strategy subsections.
Scope and taxonomy: The survey organizes efficient ViT methods into compact architecture, pruning, knowledge distillation, and quantization categories.These categories target reduced attention cost, fewer active connections, compressed teacher knowledge, and lower numerical precision, respectively.
Contributions: The paper comprehensively reviews efficient ViT methodologies, emphasizing mathematical formulations, proposed strategies, and benchmark performance comparisons.The review covers milestones through 2023 for the four selected categories.
Contributions: Efficient Error Rate combines inference-relevant parameters such as parameter count, bits, FLOPs, and model size to compare efficiency against accuracy.The survey also identifies strategies that better balance EER with estimation capability and discusses future research directions.

2 BACKGROUND

Vision Transformers extract global information through self-attention, but their computational and memory costs motivate efficient strategies. This background formalizes ViT components and introduces pruning, knowledge distillation, and quantization as compression approaches.

Vision Transformers: ViT inputs are divided into patches, flattened into token sequences, and transformed through trainable patch embeddings before attention processing.Patch embedding converts an image x ∈ R^H×W×C into a sequence of entities used by subsequent transformer blocks.
Vision Transformers: Self-attention aggregates information across the complete input sequence through query, key, value, and softmax computations.Multi-head self-attention runs several self-attention layers in parallel to extract information from different areas.
Vision Transformers: Self-attention time and memory costs increase as O(n^2) with the number of image patches because of its matrix multiplications.The two dot products contribute O(n^2d_k) and O(n^2d_v) costs, respectively.
Pruning: Pruning zeros selected trained weights after training to create lighter sparse models with fewer neurons, connections, and inference multiplications.The selection uses importance scores, while strategies differ in structure, scoring, and training or fine-tuning procedures.
Knowledge Distillation: Knowledge distillation transfers a pretrained teacher’s output distribution to a smaller student using soft-label and ground-truth cross-entropy objectives.The student learns information beyond the ground-truth labels while minimizing its distance from the teacher’s predictions.
Quantization: Quantization converts 32-bit floating-point weights and activations into lower-bit representations using a quantization function with a chosen scaling interval.Uniform quantization equally splits the data range, while the bit width and scaling factors require careful selection.

3 EFFICIENT VISION TRANSFORMER

The survey reviews efficient ViT architectures that reduce self-attention cost through spatial reduction, local attention, pooling, softmax alternatives, and other module-level changes.

Compact architecture: PVT reduces computational and memory requirements for high-resolution features by replacing standard multi-head attention with spatial-reduction attention.The spatial dimension is reduced before attention, with the reduction ratio varying across stages.
Compact architecture: Swin Transformer achieves linear complexity with input image size by restricting self-attention to non-overlapping local windows.Windowed attention avoids the global self-attention operation used in the original ViT.
Compact architecture: Linear computational complexity to token sequence length is achieved by PoolFormer’s pooling operation without adding learnable parameters.PoolFormer replaces self-attention with non-parametric pooling in a MetaFormer structure.
Compact architecture: MViTv2 limits attention complexity and memory requirements by adding a pooled query tensor as a residual connection inside the attention module.PVTv2 similarly uses average pooling for spatial reduction and overlapping patch embedding to preserve local continuity.
Compact architecture: SimA adapts computation at test time to be linear in either token count or channel count by replacing softmax with l1-normalization.Removing exp(.) operations improves efficiency on edge and low-power devices.

4 EFFICIENT VISION TRANSFORMER PERFOR-

The survey compares compact architectures, pruning, knowledge distillation, and quantization on ImageNet1K using accuracy, hardware-relevant efficiency metrics, and the Efficient Error Rate (EER). Across categories, the best trade-offs vary by deployment constraint, while Castling-MViTv2-T provides the overall accuracy–EER Pareto optimum.

4.1 Results of Compact Architectures strategies: For hardware-constrained devices, EfficientViT-B1 and Castling-LeViT-128 use 0.52G and 0.49G FLOPs, with 79.4% and 79.6% Top-1 accuracy and EER values of 6.7% and 7.4%.These models are identified as viable for scenarios requiring real-time inference.
4.1 Results of Compact Architectures strategies: Castling-MViTv2-T achieves 84.1% Top-1 accuracy and the overall accuracy–EER Pareto optimum, outperforming similar-EER compact models by an average 1.8%.It is 1.2% below heavier MViTv2-L but has a 9.2× EER gain.
4.3 Results of Knowledge Distillation strategies: TinyViT-21M provides 83.1% Top-1 accuracy with EER 24.3%, and no compared student design combines a higher classification score with similarly restricted EER.Mini-DeiT-Ti remains an option for extremely stringent computational constraints because of its very low EER.
4.4 Results of Quantization strategies: Extreme 4-bit quantization often harms small models, while 6/8-bit approaches preserve better accuracy; heavier Swin models can retain good performance even at 4 bits.For ViT-T and DeiT-Ti, APQ-ViT obtains 17.6% and 47.9% Top-1 accuracy at 4 bits, versus up to 74.8% and 72.0% at higher precision.
4.4 Results of Quantization strategies: NoisyQuant-PTQ4ViT-Swin-B reaches 84.7% Top-1 accuracy with a 66.0MB model, exceeding ViT-B and DeiT-B by 2.3% and 4.0% at nearly equal EER and model size.The identified strategy uses 6-bit quantization and is the most efficient quantized model in the comparison.

5 DISCUSSIONS AND CONCLUSION

The survey classifies efficient ViT strategies, introduces EER for comparing efficiency against a fixed baseline, and identifies limitations involving task coverage, metrics, and strategy combinations.

The survey classifies efficient ViT methods into compact architecture, pruning, knowledge distillation, and quantization, then compares them against a fixed non-efficient baseline.
Efficient Error Rate (EER) provides a general efficiency overview by comparing reviewed models with a predefined fixed baseline.
Many reviewed approaches are evaluated mainly on classification, so their behavior on different or more complex tasks remains insufficiently explored.
Common metrics such as accuracy, FLOPs, and trainable parameters do not fully capture efficiency trade-offs for real-world deployment.
Most studies focus on individual techniques, while combining multiple efficient methodologies remains a promising direction for limiting accuracy loss.

6 SUPPLEMENTARY MATERIAL

The supplementary material extends the survey with dynamic inference analyses, flexible EER formulations, and CA evaluations across detection and segmentation tasks. It also compares accuracy–efficiency trade-offs using figures and benchmark tables.

6.1 Dynamic inference methodologies: Dynamic inference methods reduce computation by adapting token processing to inputs, including halting, packaging less informative tokens, or merging redundant tokens.A-ViT discards tokens after halting conditions, SPViT packages less informative tokens, and ToMe merges redundant tokens without retraining.
6.2 Applicability and flexibility of the ERR metric: EER combines heterogeneous efficiency parameters into one value to support broad comparisons under application-specific resource constraints.The metric can incorporate parameters such as model size, FLOPs, and trainable parameters, with weights when some constraints matter more.
6.3 CA strategy COCO and ADE20K datasets: On COCO, MViTv2-T with Mask R-CNN provides the best encoder–decoder trade-off, with APbox and APmask decreases of −3.6 and −2.4 versus MViTv2-L.
6.3 CA strategy COCO and ADE20K datasets: On ADE20K, Ortho-S with Semantic FPN achieves mIoU= 48.2 using 28M trainable parameters, while larger decoder parameter counts do not yield noticeable mIoU improvement.
Supplementary comparisons: Figures 5–7 visualize accuracy–EER distributions for compact architecture, pruning, and distillation methods, highlighting models with the best trade-offs.For Figure 5, bubble area represents trainable parameter count and the plot covers ImageNet-1K Top-1 accuracy and EER.