PaperlayerView exact revision on arXiv

Source-linked AI summary

Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies

Shaibal Saha, Lanyu Xu

arXiv:2503.02891v3cs.CVcs.AR

TL;DR

ViTs’ substantial computational and memory demands hinder direct deployment on resource-constrained edge devices, while a comprehensive review of compression and acceleration trade-offs has been lacking. This survey analyzes compression methods, inference software, and hardware acceleration, finding that these techniques reduce overhead and improve inference speed while hardware adaptability, memory bottlenecks, and optimal compression remain open challenges.

  • Problem

    ViT models’ computational and memory requirements make direct deployment on resource-constrained edge devices impractical, and a comprehensive review of compression and acceleration trade-offs has been lacking.

  • Method

    The survey analyzes ViT pruning, quantization, knowledge distillation, software inference tools, and software-hardware co-design across edge platforms.

  • Results

    The analysis indicates that current ViT compression and acceleration techniques reduce computational overhead and improve inference speed, while hardware adaptability, memory bottlenecks, and optimal compression strategies remain insufficiently explored.

  • Takeaways & Limitations

    Future research should investigate hardware-aware neural architecture search, sparsity-aware accelerators, and efficient cross-platform software-hardware co-design frameworks.

  • Takeaways & Limitations

    Most edge hardware requires uniform bit-widths across operations, creating compatibility issues for mixed-precision quantization and complicating deployment.

Abstract

from arXiv · show

In recent years, vision transformers (ViTs) have emerged as powerful and promising techniques for computer vision tasks such as image classification, object detection, and segmentation. Unlike convolutional neural networks (CNNs), which rely on hierarchical feature extraction, ViTs treat images as sequences of patches and leverage self-attention mechanisms. However, their high computational complexity and memory demands pose significant challenges for deployment on resource-constrained edge devices. To address these limitations, extensive research has focused on model compression techniques and hardware-aware acceleration strategies. Nonetheless, a comprehensive review that systematically categorizes these techniques and their trade-offs in accuracy, efficiency, and hardware adaptability for edge deployment remains lacking. This survey bridges this gap by providing a structured analysis of model compression techniques, software tools for inference on edge, and hardware acceleration strategies for ViTs. We discuss their impact on accuracy, efficiency, and hardware adaptability, highlighting key challenges and emerging research directions to advance ViT deployment on edge platforms, including graphics processing units (GPUs), application-specific integrated circuit (ASICs), and field-programmable gate arrays (FPGAs). The goal is to inspire further research with a contemporary guide on optimizing ViTs for efficient deployment on edge devices.

1. Introduction

Vision Transformers model images as patch sequences and provide global context through self-attention, but their substantial computational and memory demands hinder deployment on resource-constrained edge devices. This survey reviews compression, software, and hardware-aware strategies for improving ViT efficiency across edge platforms.

  • ViTs process images as patch-embedding sequences and use self-attention to model global context, unlike CNNs’ hierarchical feature extraction.
  • ViT-Huge exceeds 632M parameters and has been extended to 22B, making direct deployment on resource-constrained edge devices impractical without optimization.
  • Model compression reduces computational overhead and redundancy, while hardware-aware acceleration addresses self-attention’s quadratic complexity and patch-embedding inefficiencies.
  • The survey systematically categorizes pruning, quantization, knowledge distillation, software tools, evaluation metrics, and hardware acceleration for GPUs, CPUs, FPGAs, and ASICs.
  • The literature search covered Google Scholar, IEEE Xplore, arXiv, and ACM Digital Library, identifying 170 papers published through January 2025.

2. Model Compression

Model compression supports ViT deployment on edge devices with limited computational power and memory while maintaining accuracy. The section focuses on pruning, knowledge distillation, and quantization, though ViT compression still requires extensive exploration.

  • Model compression enables ViT deployment on edge devices by reducing inference latency, memory use, and energy consumption while maintaining accuracy.
  • The section examines pruning, knowledge distillation, and quantization as prominent compression techniques for ViT models.
  • ViT compression still needs extensive exploration because of the architecture’s complexity and high resource-use tendencies.

2.1. Pruning

Pruning reduces ViT computation by removing redundant weights or larger structural components, with methods differing in granularity, timing, adaptability, and hardware compatibility. Recent approaches span importance-based, token, structural, cascade, and other strategies, while evaluation shows efficiency gains alongside remaining training-cost and generalizability challenges.

  • Pruning fundamentals: Pruning removes redundant parameters or connections to reduce memory and bandwidth, but unstructured sparsity may not translate into hardware efficiency.Structured pruning produces more hardware-friendly sparsity patterns, though it can incur higher accuracy costs.
  • Pruning types: Static pruning fixes pruning decisions offline, whereas dynamic pruning adapts component selection at runtime or during training.Cascade pruning combines predefined pruning stages with iterative criteria that can adapt to changing data, tasks, or hardware constraints.
  • Pruning types: ViT pruning methods are categorized into importance-based, token, structural, cascade, and miscellaneous approaches, with some methods spanning multiple categories.Patch Slimming and CP-ViT combine importance-based and structured token pruning, while CP-ViT and VTC-LFC also use cascade strategies.
  • Pruning techniques: Importance-based methods prune channels, heads, blocks, or patches using scores and thresholds, targeting computationally costly MSA and MLP modules.VTP applies L1 sparse regularization and threshold-based channel pruning, while WDPruning uses masking, head pruning, and linear-projection pruning.
  • Pruning techniques: SP-ViT introduces latency-aware soft token pruning for vanilla DeiT and Swin Transformer models.The framework is presented as a token-pruning workflow for reducing computation in ViT models.
  • Evaluation and discussion: NViT maintains competitive Top-1 (%) accuracy across several ViT models, while UVC achieves the highest GFLOPS reduction for DeiT-T/S variants.The survey also identifies limited cross-architecture evaluation and substantial training and finetuning costs as barriers to edge deployment.

2.2. Knowledge Distillation

Knowledge distillation transfers information from larger teacher models to compact student ViTs, with logit-, feature-, and relation-based approaches differing in the information they align. Across classification, detection, and segmentation, distillation generally improves student performance, although methods vary in supervision depth and task coverage.

  • Knowledge Distillation: KD trains a smaller student model to reproduce a larger teacher’s behavior, reducing computational resources while helping recover accuracy lost through other compression techniques.The survey contrasts KD with pruning: KD trains a new compact model, while pruning directly removes complexity from the original model.
  • KD Taxonomy: ViT KD methods comprise logit-based, relation-based, and feature-based approaches that transfer output predictions, inter-sample relationships, or intermediate representations.Feature-based KD aligns hidden features, whereas relation-based KD models relationships across the dataset.
  • Logit-based KD: Logit-based KD aligns final predictions but neglects intermediate features and attention, limiting effectiveness when teacher and student architectures differ substantially.DearKD combines response-based and representational distillation and surpassed a baseline trained on ImageNet-21K using 50% of the data, while some logit-based methods show smaller gains or degradation.
  • Relation- and Feature-based KD: Relation-based KD captures inter-sample relationships, while feature-based KD transfers intermediate information such as attention and hidden states.Co-advise uses multiple teachers with diverse inductive biases, and MiniViT outperforms manifold distillation in reported classification comparisons.
  • Reported Outcomes: KD improves student ViT performance across classification, object detection, and segmentation, with gains reported in top-1 accuracy, APbox, and mIoU.The survey notes that classification has received substantially more KD research than detection and segmentation.

2.3. Quantization

Quantization reduces numerical precision to lower memory use and accelerate ViT inference, with techniques organized by scheme, training requirement, calibration, granularity, and precision strategy. PTQ is prominent for edge deployment, but ViT-specific attention behavior and limited edge validation remain important challenges.

  • Quantization Overview: Quantization reduces data bit-width to save memory, accelerate inference, and simplify computation for resource-constrained edge devices.The survey categorizes quantization by timing, scheme, calibration, granularity, and precision strategy.
  • Quantization Approaches: QAT adapts models to low-precision representations through retraining, whereas PTQ quantizes fully trained floating-point models without additional training but typically causes a larger accuracy drop.PTQ is more resource-efficient, which makes it widely used for ViT edge deployment.
  • ViT Quantization Challenges: ViT quantization is challenging because quantization loss can significantly affect self-attention and because different layers learn different input features.The survey describes uniform and non-uniform schemes, dynamic and static calibration, layer-wise and channel-wise granularity, and fixed-, mixed-, and hardware-aware precision.
  • PTQ Techniques: PTQ4ViT combines twin uniform quantization with a Hessian-guided metric, while APQ-ViT uses block-wise evaluation to address Hessian-based errors in ultra-low-bit quantization.The Hessian-guided metric may miss key errors at 2-bit or 4-bit precision, motivating refined error modeling.
  • PTQ Techniques: NoisyQuant modifies activation distributions with noisy biases and reports higher top-1 accuracy than PTQ4ViT on ViT-B, DeiT-B, and Swin-S.The method removes the noisy-bias effect after activation-weight multiplication using a denoising bias.
  • Efficient Inference: Integer-only quantization and log2-based reparameterization reduce activation-processing overhead and make inference suitable for edge devices.The approach applies log2-based quantization to post-softmax activations during inference.
  • Discussion: Most reported ViT quantization studies use PTQ, while many QAT methods lack experimental validation on edge devices and rely mainly on ImageNet-1k and COCO 2017.The survey identifies specialized domains such as medical imaging and autonomous driving as underexplored for quantization generalizability.

3. Tools for Efficient Edge Deployment

Efficient ViT edge deployment combines software tools, hardware-specific optimization, and metrics for latency, throughput, power, energy, accuracy, and resource utilization. Tool choice depends on the target architecture: some tools specialize in GPUs or FPGAs, while ONNX Runtime provides broader cross-platform support.

  • Deployment Tooling: Edge deployment uses software tools for model optimization, compilation, synthesis, inference runtime, and hardware-specific execution across heterogeneous platforms.The survey frames deployment as requiring coordinated software, evaluation tools, and optimization techniques.
  • FPGA Toolchains: Vivado and Quartus synthesize FPGA designs, while Vitis AI and OpenVINO provide hardware-optimized operators and runtime environments but remain tightly coupled to their respective ecosystems.This coupling limits cross-platform portability.
  • Platform-Specific Tools: TensorRT and Triton target NVIDIA GPU deployments, Vitis AI supports customized FPGA acceleration, and ONNX Runtime spans CPUs, GPUs, and FPGAs through execution providers.oneDNN is optimized for Intel CPUs, integrated GPUs, and ARM processors, with experimental support for additional architectures.
  • Evaluation Tools: Latency and throughput evaluations use platform-specific profilers and benchmarks, including PyTorch CUDA events, TensorRT Profiler, Vitis AI Profiler, and OpenVINO benchmark tools.These tools support measurements across NVIDIA GPUs, AMD FPGAs, Intel CPUs, and Intel FPGAs.
  • Power and Energy Metrics: Power and energy evaluation uses tools such as NVIDIA-SMI, tegraStats, and Xilinx Power Estimator, with GOP/J and FPS/W serving as energy-efficiency measures.Energy estimation is more reliable on edge devices than on general-purpose platforms because background processes interfere less with readings.
  • Hardware Optimization: FPGA acceleration uses BRAM-based on-chip storage, pipeline parallelism, and loop unrolling to reduce costly memory transfers and increase throughput.Huffman coding can compress weights, while on-chip storage reduces off-chip accesses that are slow and power-hungry.

4. Accelerating Strategies For ViT on Edge

The survey’s acceleration discussion covers non-linear operations and state-of-the-art ViT techniques, evaluating them by both hardware efficiency and accuracy. Its focus is therefore on jointly assessing deployment performance and model quality.

  • Acceleration Scope: Acceleration strategies for edge ViTs include techniques for non-linear operations and broader state-of-the-art hardware acceleration methods.The section explicitly considers operations such as softmax, GELU, and LayerNorm elsewhere in the survey structure.
  • Evaluation Focus: The survey analyzes acceleration techniques using both hardware efficiency and accuracy as evaluation dimensions.This framing connects deployment cost with model-performance preservation.

4.1. Accelerating Non-linear operations

Quantized ViTs require specialized treatment of non-linear operations such as softmax, GELU, and LayerNorm, because low-bit units primarily target linear computation. Integer-only approximations and customized ASIC designs address these operations while targeting efficient inference and accuracy preservation.

  • Non-linear functions such as softmax, GELU, and LayerNorm remain critical optimization targets because low-bit units primarily support linear operations.These functions are essential components of ViT architectures and complicate quantized execution.
  • FQ-ViT uses integer-only approximations for softmax and LayerNorm through polynomial, log2, and integer-arithmetic techniques.Its softmax approximation uses a second-order polynomial with log2 quantization, while LayerNorm uses PTF-based integer computation of mean and variance.
  • PackQViT replaces e with 2 in softmax using second-order polynomial approximations, while EdgeKernel optimizes bit-shift selection on Apple A13 and M1 chips.PackQViT requires training and reports no accuracy loss; EdgeKernel targets precision preservation with reduced bit truncation.
  • ASIC-oriented methods address non-linear operations through software and hardware specialization, including E2Softmax, PTF-based LayerNorm, and customized FP32 units.SOLE emphasizes software optimization, whereas SwiftTron develops customized ASIC hardware for non-linear operations.

4.2. Current Accelerating Techniques on ViT

Current ViT acceleration combines software, hardware, and SW-HW co-design techniques to reduce computation, latency, and energy consumption across edge platforms. The survey highlights quantization, pruning, adaptive computation, custom hardware, and platform-specific implementations, while noting trade-offs in flexibility and efficiency.

  • SW-HW co-design: SW-HW co-design integrates algorithmic and hardware-aware optimizations to balance throughput, latency, energy efficiency, and accuracy.The surveyed approaches include quantization, sparse or adaptive attention, mixture-of-experts layers, kernel profiling, scheduling, and custom non-linear units.
  • Quantization-Based Software Acceleration: VAQF automatically selects quantization parameters from model structure and target FPS, using binary weights and low-precision activations for hardware-efficient inference.Its stated objective is high throughput while maintaining model accuracy.
  • Pruning-Based Software Acceleration: Sparse attention and token pruning reduce redundant computation caused by self-attention’s quadratic sequence-length complexity.ViTCoD uses structured pruning of attention scores, while HeatViT removes unimportant tokens before transformer blocks.
  • Other Approaches: Mixture-of-experts improves efficiency by routing each input to only a subset of expert networks, reducing unnecessary computation.M3ViT uses a dynamic router to select relevant experts while balancing model capacity and efficiency.
  • Hardware Optimization in SW-HW Co-design: Hardware implementations use C++ descriptions, HLS, bitstream generation, AIE kernels, and custom units to optimize edge execution.These mechanisms connect software choices with accelerator-specific execution requirements.
  • ASIC-Based Hardware Acceleration: ViT-specific ASIC accelerators target attention and normalization demands that differ from CNN accelerator requirements.ViTCoD and SOLE are cited as SW-HW co-designed ASIC implementations for edge acceleration.
  • FPGA-Based Hardware Acceleration: FPGA methods combine quantization, resource modeling, parallelism tuning, HLS synthesis, and bitstream generation for deployment.VAQF and Auto-ViT Acc adjust accelerator parameters when placement or routing constraints limit parallelism.
  • Discussion: FPGAs and ACAPs often provide customized, lower-latency, energy-efficient execution, whereas GPUs and EdgeGPUs leverage broad parallel processing.The survey associates platform choice with distinct optimization levels and deployment characteristics.

4.3. Performance Analysis for Accelerating Techniques

The survey compares accelerator performance through resource use, energy efficiency, throughput, and accuracy across edge platforms and ViT tasks. Results show substantial hardware-dependent trade-offs, including speedup versus energy efficiency and accuracy versus energy consumption.

  • Resource Utilization: Resource utilization varies by accelerator design, with DSP, BRAM, KLUT, and KFF usage reflecting different optimization strategies and deployment constraints.The analysis treats resource use as a measure of hardware efficiency.
  • Resource Utilization: HeatViT uses 2066 DSPs and Zhang et al. uses 2147 DSPs, while ViA uses 1002 BRAMs and VAQF uses 565.5 BRAMs.EQ-ViT uses 16 BRAMs, indicating a markedly different on-chip memory strategy.
  • Energy Efficiency: On the same AMD ZCU102 FPGA board and matched frequency and resources, VAQF outperforms Auto-ViT-Acc and HeatViT in energy and power usage.The survey cautions that cross-device energy comparisons are difficult and reports no original-paper energy comparison for ViTCoD or SOLE.
  • Throughput: Zhang et al. achieves 2330.2 GOPs throughput, compared with 1181.5 GOPs for Auto-ViT-Acc and 1217.4 GOPs for M3ViT.Throughput varies with hardware architecture, precision, and optimization strategy; some studies do not report it.
  • Accuracy: Auto-ViT-Acc reaches 81.8% Top-1 accuracy on DeiT-Base, surpassing VAQF’s 77.6% for classification.For DeiT-Tiny, EQ-ViT reaches 74.5%, while HeatViT, ViTCoD, and SOLE report 72.1%, 70.0%, and 71.07%, respectively.
  • Accuracy: EQ-ViT’s higher DeiT-Tiny accuracy is accompanied by significantly higher energy consumption, illustrating an accuracy-efficiency trade-off.The comparison covers Top-1 accuracy on ImageNet-1K and mIoU for segmentation benchmarks.
  • Accuracy: M3ViT achieves 72.8 mIoU on PASCAL-Context and 45.6 mIoU on NYUD-v2, indicating dataset-sensitive segmentation performance.Its evaluation extends beyond classification to segmentation benchmarks.

5. Challenges and Future Directions of ViT on Edge Devices

Edge deployment requires coordinated software–hardware design because compression alone may not deliver real-time performance across heterogeneous platforms. Future directions include hardware-aware search, sparsity-aware acceleration, mixed-precision support, and compact mathematical representations.

  • Edge AI can improve ViT accessibility while reducing latency, privacy risks, and operational costs, but real-world deployment remains challenging.
  • SW-HW co-design: Hardware platforms differ in precision, memory bandwidth, and computational efficiency, making hardware-aware compression and reconfigurable software–hardware co-design necessary.Existing frameworks often cannot handle sparsity introduced by model compression.
  • NAS for compression search: NAS methods automate pruning and quantization but remain hardware-specific, and extending hardware-aware search to ViTs is still an open challenge.ProxylessNAS predicts latency effectively but is limited to CNN architectures.
  • NAS for compression search: Future NAS research could jointly search quantization bit-widths, pruning ratios, and distillation strategies against latency, memory, and energy objectives.
  • Sparsity and mixed precision: Irregular sparse tensors and mixed-precision operations create latency, energy, compatibility, and bit-padding overheads on current processors.
  • Sparsity and mixed precision: Sparsity-aware accelerators, bit-level reconfigurable units, and tensor-train decomposition are proposed to reduce unnecessary computation and compact ViT representations.Suggested mechanisms include zero-skipping, compressed sparse row encoding, and low-rank weight formats.

5.4. Automated Edge Aware Model Compression

Automated, hardware-aware compression remains necessary because current ViT optimization often depends on manual settings, narrow benchmarks, and post-training procedures. Broader evaluation must account for task, modality, diagnostic, and real-world deployment requirements.

  • Most compression methods require manual tuning of bit width, pruning ratio, or layer-wise sparsity within a resource budget.Only limited work explores automatic adjustment of compression parameters.
  • Most current ViT compression work is post-training, while compression during training remains costly and underexplored beyond promising quantization-aware training.
  • Benchmarking: Benchmarking remains concentrated on ImageNet-1k classification and COCO-2017 detection, limiting evidence across real-world application areas.The survey calls for additional datasets and models, including medical imaging and autonomous driving.
  • Real-world evaluation: Compression, acceleration, and framework evaluations remain largely academic, with few studies testing ViT deployment in practical settings such as medical imaging.
  • Real-world evaluation: Medical imaging compression must preserve spatial resolution and feature details because small accuracy losses can affect diagnostic outcomes.Transfer learning also complicates which pretrained layers should be retained or modified.

5.7. Seamless Model-to-Edge Integration

Moving trained ViTs onto edge hardware remains labor-intensive, tool support is still immature for ViTs, and heterogeneous data can make a universal compression strategy unreliable.

  • Converting a trained model into a hardware-compatible inference version requires substantial time, cost, manual labor, and combined software–hardware expertise.Available automatic mapping tools remain insufficient, particularly for ViTs.
  • Current edge frameworks support rapid prototyping but often lack mature ViT support and have primarily been evaluated on CNN models.FINN-R provides FPGA quantization support but only for standard techniques.
  • Sensor, vendor, and medical-imaging modality heterogeneity means a compression technique effective for one data source may not transfer reliably to another.

6. Conclusion

The survey synthesizes ViT compression and hardware acceleration methods for edge deployment, while identifying unresolved hardware adaptability, memory, and compression-strategy challenges. It frames NAS, sparsity-aware accelerators, and cross-platform software–hardware co-design as future directions.

  • The survey analyzes pruning, quantization, knowledge distillation, software tools, and software–hardware co-design for ViT edge deployment.
  • Hardware adaptability, memory bottlenecks, and optimal compression strategies remain unresolved despite progress in reducing computational overhead and improving inference speed.
  • The survey aims to support continued innovation in an evolving field of ViT optimization for edge devices.
Loading 2503.02891v3…