Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies

1. Introduction

• ViTs deliver SOTA performance across classification, detection, and segmentation by processing image patches with self-attention, but their large computational and memory demands hinder edge deployment [4].ViT-Huge exceeds 632M parameters and has been extended to 22B parameters, making direct deployment impractical without optimization.Source: §1 · p. 2
• Compression methods including pruning, quantization, and knowledge distillation reduce ViT size, computation, latency, and power consumption, but often cannot alone satisfy stringent real-time edge requirements.The survey frames compression as necessary for reducing redundancy and improving efficiency, while noting that hardware constraints still require additional acceleration.Source: §1 · p. 2, Figure 1 · p. 3
• Effective ViT deployment therefore combines hardware-aware optimization, efficient nonlinear operations, specialized accelerators, optimized libraries, and software-hardware co-design across GPUs, ASICs, and FPGAs.These strategies target self-attention’s quadratic complexity and patch-embedding inefficiencies to support low-latency inference in applications such as autonomous driving, mobile vision, and medical imaging.Source: §1 · pp. 2–3

1.1. Motivations and Contribution

• The survey addresses the lack of a unified edge-focused resource by systematically analyzing compression, inference software, evaluation metrics, hardware acceleration, comparative trade-offs, and future research directions [18].Its contributions cover pruning, quantization, knowledge distillation, efficient attention, software-hardware co-design, FPGA optimization, and deployment across GPUs, CPUs, FPGAs, and ASICs.Source: §1 · p. 3, §1.1 · p. 4

1.2. Literature Collection and Organizations

• The literature review collected 170 papers published through January 2025 from major academic databases, selecting studies on ViT compression, cross-platform acceleration benchmarks, and hardware-aware or software-hardware co-design strategies.The search used keywords covering vision transformers, acceleration, edge devices, co-design, pruning, quantization, and knowledge distillation.Source: §1.2 · pp. 4–5

2. Model Compression

• Compression reduces ViT inference latency, memory use, and energy consumption while targeting accuracy preservation for edge deployment.Its application to ViTs still requires extensive exploration because their complex architectures tend to have high resource usage.Source: §2 · p. 5
• The section surveys three prominent ViT compression techniques: pruning, knowledge distillation (KD), and quantization.These techniques are discussed in Sections 2.1, 2.2, and 2.3, respectively.Source: §2 · p. 5

2.1. Pruning

• Pruning removes redundant connections to reduce memory, bandwidth, parameters, and computation, producing models with fewer edges than the original network.Early approaches used biased weight decay, second-order derivatives, channel methods, and Hessian-based criteria [31], often requiring manual identification of weights that preserve accuracy.Source: §2.1 · pp. 5–6

2.1.1 Types

• Structured pruning produces hardware-friendly sparsity by removing layers, neurons, or channels, whereas unstructured pruning removes individual weights but may not improve accelerator efficiency.A two-stage coarse-grained/fine-grained structured method based on top-K sparsification reduced 60% overall computation in embedded neural networks.Source: §2.1.1 · p. 7
• Static pruning uses a fixed offline schedule suited to known hardware constraints, while dynamic pruning adapts runtime criteria to changing component importance and can preserve accuracy.Magnitude-based static pruning removes the smallest weights under a specified rate r; recent methods commonly use dynamic pruning [36].Source: §2.1.1 · pp. 7–8

2.1.2 Pruning Techniques for Vision Transformer

• Cascade pruning iteratively combines dynamic and structured decisions, while SP-ViT performs latency-aware soft token selection and VTC-LFC progressively prunes tokens and channels under a performance-drop constraint.SP-ViT consolidates less informative tokens instead of discarding them and has been extended beyond classification to detection and segmentation; cascade ViT pruning [36] targets patch-head regions in MSA and FFN.Source: §2.1.1 · p. 8, §2.1.2 · pp. 10–11
• ViT pruning methods include importance-based, token, structure, cascade, and miscellaneous approaches, often overlapping across categories and targeting dimensions, heads, channels, blocks, or patches.VTP [38] uses L1 sparse regularization, thresholded channel pruning, and fine-tuning; WDPruning combines width and depth pruning, while patch slimming removes redundant patches layer by layer.Source: §2.1.2 · pp. 8–10

2.1.3 Discussion

• NViT maintained competitive Top-1 accuracy across DeiT-T/S/B and Swin-S, while UVC achieved the highest GFLOPs reduction for DeiT-T/S.NViT reduced DeiT-B GFLOPs while improving Top-1 accuracy, and achieved greater parameter reduction than other ViT models; the comparison covers ImageNet-1K except CP-ViT [36], evaluated on CIFAR.Source: Table 3 · p. 12, §2.1.3 · p. 13, Table 4 · p. 13
• Training and fine-tuning costs remain major limitations for edge deployment, motivating training-efficient or fine-tuning-free pruning with better sensitivity estimates from limited data.Future work should also examine information embedded in hidden features during training and inference.Source: §2.1.3 · p. 14

2.2. Knowledge Distillation

• KD trains a compact student to reproduce a larger teacher’s behavior and can recover accuracy lost through pruning or quantization during finetuning.Unlike pruning, KD creates a new deployable model rather than directly simplifying the original network.Source: §2.2 · p. 14

2.2.1 KD techniques for Vision Transformer

• ViT KD methods are classified as logit-based, relation-based, or feature-based, transferring output predictions, inter-sample relationships, or intermediate representations.Feature-based KD aligns hidden-layer features, whereas relation-based KD models relationships across the dataset rather than only individual samples.Source: §2.2.1 · pp. 14–15, Figure 6 · p. 15
• Logit-based KD aligns final predictions but provides shallow supervision, while feature- and relation-based methods transfer richer intermediate, attention, hidden-state, spatial, or inductive-bias information.MiniViT combines prediction-logit, attention, and hidden-state losses; Co-advise uses diverse teachers, and TAT performs fine-grained spatial alignment.Source: §2.2.1 · pp. 16–18

2.2.2 Discussion

• KD techniques consistently improve student ViT performance, with relation-based Co-advise improving ImageNet-1k top-1 accuracy, MiniViT outperforming manifold distillation, and some logit-based methods such as DearKD slightly degrading accuracy.Detection KD methods improve APbox, while segmentation KD produces substantial mIoU gains; Tables 6–8 summarize results across these tasks.Source: Table 6 · p. 16, Table 7 · p. 17, Table 8 · p. 18
• Most ViT KD research targets classification, leaving comparatively fewer studies for object detection and segmentation despite their observed performance gains.This imbalance motivates further investigation for vision-driven applications requiring efficient, high-performance models.Source: §2.2.2 · p. 19

2.3.1 Taxonomy of Quantization Methods

• Quantization methods vary by timing, scheme, calibration, granularity, and precision strategy, balancing accuracy, implementation complexity, and hardware requirements.Uniform and non-uniform schemes, QAT and PTQ, dynamic and static calibration, layer-wise and channel-wise granularity, and fixed, mixed, and hardware-aware precision form the main taxonomy.Source: §2.3.1 · pp. 19–21, Table 9 · p. 20

2.3.2 Quantization Techniques for Vision Transformer

• ViT quantization research emphasizes PTQ because it avoids retraining, addressing attention-sensitive errors through statistic-based, integer-only, mixed-precision, and learning-based methods.PTQ4ViT uses twin uniform quantization and Hessian-guided scaling, while APQ-ViT introduces block-wise calibration and MPQ; I-ViT achieved 81.74% top-1 accuracy on DeiT-B with 8-bit integer-only inference, outperforming I-BERT by 0.95%.Source: §2.3.2 · pp. 22–24
• QAT for ViTs remains underexplored, using knowledge distillation or standalone frameworks to reduce information distortion, stabilize training, and support low-bit or integer-only deployment.PSAQ-ViT targets data-free quantization across classification and detection, while PackQViT combines sub-8-bit QAT, integer-only operators, and a SIMD-based 4-bit packed multiplier for mobile acceleration.Source: §2.3.2 · pp. 27, 29

2.3.3 Discussion

• PTQ methods dominate reported ViT studies, whereas QAT methods often lack edge-device validation and are mainly evaluated on ImageNet-1k and COCO 2017, limiting demonstrated generalizability.PackQViT is an exception that experiments on mobile devices; broader deployment effectiveness also depends on software toolchains and hardware-aware acceleration.Source: §2.3.3 · pp. 29–30

3.1. Software Tools

• Software tools span model optimization, compilation, high-level synthesis, and inference runtimes, with compatibility across FPGA, GPU, and CPU platforms.Vitis AI and OpenVINO support FPGA deployment but remain tied to their ecosystems, whereas ONNX Runtime provides hardware-agnostic deployment across CPUs, GPUs, and FPGAs.Source: §3.1 · pp. 30–33

3.2. Evaluation Tools

• Evaluation tools measure ViT acceleration through accuracy, latency, power consumption, energy efficiency, and resource utilization using hardware-vendor libraries and profilers.These measurements support precise assessment of performance and efficiency across edge platforms.Source: §3.2 · p. 33

3.2.1 Latency

• GPU, FPGA, and Intel-platform tools provide standardized or platform-specific latency and throughput measurements during inference deployment.PyTorch CUDA events, TensorRT Profiler, Vitis AI Profiler, and OpenVINO benchmark tools target different hardware environments.Source: §3.2.1 · pp. 33–34

3.2.2 Power

• Power evaluation uses platform-specific monitors and estimators to reveal power efficiency, thermal behavior, and performance trade-offs across CPUs, GPUs, FPGAs, and ACAPs.Intel Power Gadget, NVIDIA-SMI, tegraStats, Xilinx Power Estimator, and Vaitrace support measurement or estimation across these platforms.Source: §3.2.2 · p. 34

3.2.3 Energy

• Energy efficiency is estimated with GOP/J and FPS/W, but background processes complicate general-purpose-platform measurements compared with controlled edge-device workloads.Vivado Power Analyzer, RAPL, and combined power-and-latency measurements support energy analysis.Source: §3.2.3 · p. 34

3.2.4 Resosurce Utlization

• FPGA resource utilization is optimized and monitored through Vivado, Quartus Prime, Vitis AI, and AOCL during synthesis and acceleration evaluation.These tools help minimize resource usage and improve hardware efficiency for ViT acceleration.Source: §3.2.4 · pp. 34–35

3.3. Common Optimization Techniques

• Common FPGA optimizations include memory compression, BRAM use, pipeline parallelism, loop unrolling, layer fusion, simplified activations, optimized matrix operations, and dynamic precision.Mixed, fixed-point, and PoT precision trade accuracy against performance, while systolic arrays and linear-algebra cores accelerate transformer matrix multiplications.Source: §3.3 · p. 35
• Hardware-specific low-level acceleration remains necessary for real-time edge performance beyond software frameworks’ hardware-aware execution capabilities.Dedicated acceleration methods for edge ViTs are addressed in Section 4.Source: §3.3 · p. 35

• The section explores acceleration strategies for nonlinear operations in ViTs.Source: §4 · p. 35
• It discusses state-of-the-art techniques for accelerating ViTs on edge devices.Source: §4 · p. 35
• The performance analysis considers both hardware efficiency and accuracy.Source: §4 · pp. 35–36

4.1. Accelerating Non-linear operations

• Low-bit hardware often lacks support for softmax, GELU, and LayerNorm, forcing FP32 computation and repeated quantization/dequantization that increase latency, power, and energy consumption.Integer-based approximations target these non-linear operations to avoid frequent FP32 computations.Source: §4.1 · p. 36
• FQ-ViT uses integer softmax and LayerNorm approximations, while I-ViT adds iterative integer square-root computation and ShiftGELU, evaluated on an NVIDIA RTX 2080 Ti GPU.FQ-ViT applies LIS with a second-order polynomial and log2 quantization for softmax, and PTF for LayerNorm; I-ViT uses sigmoid-based GELU approximation.Source: §4.1 · p. 36, Table 14 · p. 37
• PackQViT simplifies softmax approximation by replacing e with 2, while EdgeKernel optimizes bit shifting on Apple A13 and M1 chips and uses uint16 LayerNorm inputs.PackQViT requires training but reports no accuracy loss; EdgeKernel uses asymmetric quantization for LayerNorm inputs to improve efficiency while preserving data integrity.Source: §4.1 · pp. 36–37
• SOLE uses E2Softmax with log2 quantization and a two-stage PTF-based LayerNorm unit, whereas SwiftTron develops customized ASIC hardware for non-linear operations, including FP32 computation.SOLE targets software optimization, while SwiftTron accommodates diverse scaling factors through customized hardware.Source: §4.1 · p. 37

4.2.1 Software Optimization in SW-HW Co-design

• Software co-design combines quantization, pruning, adaptive attention, MoE, profiling, and custom softmax or LayerNorm to balance throughput, latency, energy, and accuracy.VAQF and AutoViT-Acc use FPS-targeted PTQ, EQ-ViT uses latency- and accuracy-aware QAT with scheduling, and HeatViT prunes tokens adaptively.Source: §4.2.1 · pp. 37–39

4.2.2 Hardware Optimization in SW-HW Co-design

• Hardware co-design uses HLS, accelerator bitstreams, AIE kernels, systolic processing, buffering, and custom units to address GEMM, memory movement, nonlinear operations, and irregular MoE execution.FPGA workflows synthesize C++ descriptions through Vivado or Vitis HLS; M3ViT uses expert-wise processing and ping-pong buffering, while EQ-ViT and SOLE optimize nonlinear execution.Source: §4.2.2 · pp. 39–42
• Pure hardware approaches improve data locality and resource use through partitioning, half-layer mapping, reuse engines, broadcast MAC units, and sparse-attention processing.ViA, ViTA, and ViTCoD redesign dataflow or processing structures to reduce memory access and accelerate attention computation.Source: §4.2.2 · p. 42

4.2.3 Other Techniques

• Distributed ED-ViT [125] partitions ViTs across multiple Raspberry Pi-4B devices, while COSA Plus dynamically switches systolic-array dataflows to adapt computation and data movement.These approaches offer scalable alternatives to single accelerators by exploiting distributed workloads or runtime-configurable weight-stationary and output-stationary execution.Source: §4.2.3 · pp. 42–43

4.2.4 Discussion

• VAQF and Auto-ViTAcc use PTQ to reduce precision while preserving accuracy, whereas HeatViT and EQ-ViT optimize workloads; MoE remains difficult because irregular memory access harms utilization.Ping-pong buffering mitigates MoE memory stalls, but further optimization is needed for hardware parallelism and memory efficiency.Source: §4.2.1 · pp. 38–39, §4.2.2 · p. 41, §4.2.4 · p. 43
• ASIC designs dominate speedup, whereas FPGAs provide better energy efficiency but slightly lower absolute speedup, revealing a power–performance trade-off.Table 15 supports the platform-level comparison, while Table 17 defines the reported performance metrics.Source: Table 15 · p. 38, §4.2.4 · p. 43, Table 17 · p. 44
• Most ViT acceleration methods target specialized FPGAs or ASICs, limiting deployment flexibility; PIVOT is more adaptable because it maintains high accuracy across GPPs.ViTCoD, HeatViT, and VAQF are described as hardware-specialized alternatives, while Table 16 compares accuracy, adaptability, and compatibility.Source: §4.2.4 · pp. 43–44, Table 16 · p. 43

4.3.1 Resource Utilization

• HeatViT and Zhang et al. use the most DSPs, at 2066 and 2147 respectively, while ViA and VAQF consume 1002 and 565.5 BRAMs; EQ-ViT uses only 16 BRAMs.Auto-ViT-Acc and HeatViT require 186 and 161.4 KLUTs, respectively, reflecting logical operations for mixed-precision quantization and token pruning.Source: §4.3.1 · pp. 44–45

4.3.2 Energy Efficiency

• On the common AMD ZCU102 FPGA setup, VAQF outperforms Auto-ViT-Acc and HeatViT in energy and power usage, although ASIC energy comparisons were unavailable.Energy efficiency is measured using GOP/J or FPS/watt, and cross-platform comparisons are difficult because edge targets differ.Source: §4.3.2 · p. 45

4.3.4 Accuracy

• Auto-ViT-Acc achieves 81.8% Top-1 accuracy on DeiT-Base versus VAQF’s 77.6%, while EQ-ViT leads DeiT-Tiny at 74.5% but consumes significantly more energy.DeiT-Tiny results also include HeatViT at 72.1%, ViTCoD at 70.0%, and SOLE at 71.07%.Source: §4.3.4 · p. 46, Table 18 · p. 46
• M³ViT achieves 72.8 mIoU on PASCAL-Context but 45.6 mIoU on NYUD-v2, indicating dataset-sensitive compression performance.The accuracy analysis covers DeiT-Base, DeiT-Tiny, ViT-Base, and ViT-Small, with most methods evaluated for ImageNet-1K classification.Source: §4.3.4 · pp. 45–46, Table 18 · p. 46

5. Challenges and Future Directions of ViT on Edge Devices

• Edge deployment can improve accessibility while reducing cloud reliance, latency, privacy risks, and operational costs, although real-world scenarios and SW-HW co-design remain underexplored.Efficient, cost-effective ViT implementation directly on edge hardware is becoming possible as edge AI advances.Source: §5 · p. 46

5.1. Advancement in SW-HW Co-design

• SW-HW co-design and hardware-aware compression can address heterogeneous precision, memory-bandwidth, and computational constraints, but existing frameworks such as DNNWeaver, VAQF [122], and M3ViT generally cannot exploit compression-induced sparsity.Future pipelines should integrate compression-aware models, hardware profiling, hardware-agnostic intermediate representations, and unified compiler toolchains.Source: §5.1 · p. 47

5.2. Utilizing NAS for Inference and Compression Parameter Search

• NAS remains limited for edge ViTs because of high search costs and hardware coupling; HAQ and APQ automate compression search, whereas ProxylessNAS [138] targets CNNs rather than ViTs.Extending hardware-aware search to hierarchical, attention-based ViTs remains an open challenge.Source: §5.2 · p. 47
• Future NAS should jointly search quantization bit-widths, pruning ratios, and distillation strategies using hardware objectives such as latency, memory usage, and energy efficiency.Reinforcement learning could guide hardware-constrained searches, including FPGA custom designs requiring substantial hardware expertise.Source: §5.2 · p. 47

5.3. Hardware Accelerators to Handle Sparsity and Mixed Precision

• Current accelerators inefficiently process sparse tensors by fetching zeros, while uniform bit-width requirements create padding overhead and hinder mixed-precision ViT deployment.These limitations cause unnecessary latency, energy, and memory overhead across GPUs, CPUs, and FPGAs.Source: §5.3 · p. 48
• Sparsity-aware accelerators could dynamically skip zero computations through CSR encoding, zero-skipping logic, and sparsity-driven scheduling.Bit-level reconfigurable compute units could support mixed precision natively, optimizing storage and computation of nonzero ViT values.Source: §5.3 · p. 48
• Tensor-train decomposition can reduce ViT parameter count and computational complexity by representing high-dimensional weight matrices in compact, low-rank form.TT-ViT, TT-MLP, and hardware-aware implementations support its potential for transformer compression and FPGA deployment, though edge-specific integration remains limited.Source: §5.3 · p. 48

5.4. Automated Edge Aware Model Compression

• Automated compression should adapt quantization, pruning, and sparsity to resource budgets while minimizing accuracy degradation; SparseViT dynamically prunes less-important image regions for significant latency reductions.Hardware-efficient automaticity remains underexplored.Source: §5.4 · p. 48
• Most ViT compression remains post-training, although QAT techniques show promise; efficient training-time exploration remains limited by cost, time, and convergence demands.Broader automated compression during training could improve hardware realization.Source: §5.4 · p. 49

5.5. Developing Benchmarks

• Universal edge benchmarks should cover compression and accelerators with comprehensive metrics, since current evaluations rely mainly on ImageNet-1k and COCO-2017 [96].System-level benchmarking datasets and models remain limited beyond classification and object detection.Source: §5.5 · p. 49
• New benchmarks and ViT models are needed for application areas such as medical imaging and autonomous driving, while task-independent universal compression remains challenging.Different computer-vision tasks may require different compression strategies.Source: §5.5 · p. 49

5.6. Real-world Case Studies

• Although ViT edge deployment is advancing, compression and accelerators are rarely evaluated on real-world medical-imaging models spanning classification, segmentation, and detection.Three-dimensional ultrasounds and MRIs create distinctive deployment challenges.Source: §5.6 · p. 49
• Medical-imaging compression must preserve accuracy, latency, precision, spatial resolution, and feature details because small degradations can significantly affect diagnostic outcomes [158].Compression may discard vital low-level details needed for accurate diagnoses.Source: §5.6 · pp. 49–50
• Transfer learning further complicates medical ViT compression because selecting layers to retain or modify may lose critical learned representations.Balancing model efficiency with diagnostic reliability remains an open direction for healthcare edge applications.Source: §5.6 · p. 50

5.7. Seamless Model-to-Edge Integration

• Model-to-hardware conversion requires substantial time, cost, and manual labor at each inference-deployment step.Source: §5.7 · p. 50
• Training, compression, and acceleration require distinct software and hardware expertise, creating a knowledge gap that complicates integrated deployment.Source: §5.7 · p. 50
• FPGAs can mitigate some deployment limitations because they support reconfigurable operations and modules.Source: §5.7 · p. 50
• Available tools remain insufficient for automatically mapping models to hardware, with more limited support specifically for ViTs.Source: §5.7 · p. 50
• Current edge frameworks accelerate prototyping but inadequately support rapidly expanding architectures and remain largely CNN-focused, with ViT support still experimental.Source: §5.7 · p. 50
• FINN-R provides FPGA quantization support but is limited to standard ViT techniques, motivating budget-aware one-click deployment and automatically generated compression.Source: §5.7 · p. 50

5.8. Robustness to Diverse Data Modality

• Sensor and vendor heterogeneity, including X-rays, MRI, CT, and ultrasound, means compression effective for one modality may fail for another and threaten edge consistency.Source: §5.8 · p. 50
• Federated learning, multi-modal fusion, and adaptive data calibration are proposed to mitigate modality-driven data inconsistency at edge devices.Source: §5.8 · p. 50

• The survey comprehensively covers pruning, quantization, knowledge distillation, software tools, and software–hardware co-design for ViT compression and acceleration.Source: §6 · p. 51
• Although current techniques reduce computational overhead and improve inference speed, hardware adaptability, memory bottlenecks, and optimal compression strategies remain unresolved.Source: §6 · p. 51
• Future directions include hardware-aware NAS, sparsity-aware accelerators, and efficient cross-platform software–hardware co-design frameworks.Source: §6 · p. 51
• Optimizing ViTs for edge devices remains an evolving field with substantial challenges and opportunities for continued innovation.Source: §6 · p. 51