PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey of Techniques for Optimizing Transformer Inference

Krishna Teja Chitty-Venkata, Sparsh Mittal, Murali Emani, Venkatram Vishwanath, Arun K. Somani

arXiv:2307.07982v1cs.LGcs.ARcs.CLcs.CV

TL;DR

The survey addresses the steep memory and computation costs of increasingly large transformer models. It synthesizes inference optimizations across model compression, architecture search, lightweight design, and hardware, and summarizes tradeoffs and remaining research gaps.

  • Problem

    Transformer growth has increased memory and computation demands, creating a need for efficient deployment through compression and hardware acceleration.

  • Method

    The paper surveys inference optimizations including pruning, model compression, hardware-aware design, and accelerator techniques across transformer architectures.

  • Results

    The survey organizes optimization methods and quantitative model results to characterize tradeoffs among predictive performance, parameters, FLOPs, latency, energy, and hardware efficiency.

  • Takeaways & Limitations

    Efficient transformer inference requires considering algorithmic techniques together with hardware design and deployment constraints.

  • Takeaways & Limitations

    Current research still faces challenges in scalability, interpretability, and fair comparison across optimization techniques.

Abstract

from arXiv · show

Recent years have seen a phenomenal rise in performance and applications of transformer neural networks. The family of transformer networks, including Bidirectional Encoder Representations from Transformer (BERT), Generative Pretrained Transformer (GPT) and Vision Transformer (ViT), have shown their effectiveness across Natural Language Processing (NLP) and Computer Vision (CV) domains. Transformer-based networks such as ChatGPT have impacted the lives of common men. However, the quest for high predictive performance has led to an exponential increase in transformers' memory and compute footprint. Researchers have proposed techniques to optimize transformer inference at all levels of abstraction. This paper presents a comprehensive survey of techniques for optimizing the inference phase of transformer networks. We survey techniques such as knowledge distillation, pruning, quantization, neural architecture search and lightweight network design at the algorithmic level. We further review hardware-level optimization techniques and the design of novel hardware accelerators for transformers. We summarize the quantitative results on the number of parameters/FLOPs and accuracy of several models/techniques to showcase the tradeoff exercised by them. We also outline future directions in this rapidly evolving field of research. We believe that this survey will educate both novice and seasoned researchers and also spark a plethora of research efforts in this field.

I. INTRODUCTION

Transformer capabilities and applications have expanded rapidly, but increasing model size has sharply raised memory and computation costs. This survey organizes inference optimizations across algorithms, architectures, and hardware while highlighting their efficiency–predictive-performance tradeoffs.

  • Motivation: State-of-the-art language models can reach 1.2 trillion parameters, while vision transformers have scaled to 22 billion parameters.These sizes increase memory and computation overheads; ChatGPT inference is also associated with substantial water consumption.
  • Optimization landscape: Efficient inference methods target transformer size, latency, and energy through pruning, quantization, knowledge distillation, neural architecture search, and hardware-aware design.Hardware optimization includes mapping models to FPGAs and ASICs and avoiding redundant or ineffectual computation.
  • Scope: The survey covers inference-related optimization techniques for BERT, GPT, ViT, and related transformer architectures while excluding training-related techniques.Its stated goals are reducing inference time, minimizing memory requirements, and enhancing hardware performance.
  • Hardware considerations: Hardware-unaware compression may reduce model size without reducing latency, memory accesses, or energy because irregular sparsity can prevent vectorization and tiling.Approximate computing can likewise require fewer operations yet incur higher GPU latency.
  • Survey contributions: The paper provides a taxonomy organized by optimization technique, model granularity, transformer architecture, and application domain.It combines qualitative discussion with quantitative results on parameters, FLOPs, and accuracy to expose efficiency tradeoffs.
  • Future directions: The survey identifies future directions including efficient hardware architectures, algorithm–hardware co-design, combined optimization techniques, and novel benchmarks.These directions are framed as responses to gaps in current inference optimization research.

II. BACKGROUND ON TRANSFORMER NETWORKS

Transformer networks combine stacked encoder and decoder layers with multi-head attention and feed-forward processing. Self-attention forms pairwise relationships among sequence elements, while positional information and residual normalization support sequence modeling.

  • Transformer architecture: The vanilla transformer contains stacked encoder and decoder modules built from multi-head attention, feed-forward networks, normalization, and residual connections.The encoder processes an input sequence into a representation, while the decoder uses encoder context to generate an output sequence.
  • Input representation: Positional information is added because transformers lack recurrence and convolution, using sine and cosine functions at alternating positions.The embedding layer converts tokens into dense vectors that are then supplied to attention.
  • Self-Attention: Self-attention projects inputs into Query, Key, and Value tensors, computes scaled Query–Key scores, applies softmax, and combines the resulting attention with Values.Dividing by √D_k is described as alleviating gradient vanishing, while softmax amplifies high scores and suppresses lower ones.
  • Multi-Head Self-Attention (MHA): Multi-head attention computes self-attention concurrently across multiple heads, then concatenates and linearly transforms their independent outputs.Each head receives the input through separate fully connected projections for Query, Key, and Value.
  • Feed-Forward Network: The feed-forward network uses two fully connected layers with ReLU or GELU activation to learn position-specific information after attention processing.Its output is further processed through normalization within the transformer layer.

2) Decoder:

The decoder uses masked self-attention, cross-attention, and a feed-forward network to generate sequences while preventing access to future positions. The surrounding section also introduces BERT, GPT, and ViT architectures and their core processing structures.

  • 2) Decoder:: The decoder stacks masked MHA, encoder–decoder cross-attention, and an FFN to generate output sequences.Masked MHA prevents access to future positions, while cross-attention combines encoder output with the generated sequence.
  • BERT: BERT uses only the encoder, masks 15% of input words, and is configured by encoder layers, hidden size, and attention heads.Its bidirectional predictions use both preceding and following words.
  • GPT: GPT retains the decoder with positional encoding, masked MHA, FFN, and normalization for predictive language tasks.GPT variants include GPT-1, GPT-2, and GPT-3, with applications such as ChatGPT.
  • ViT: ViT converts images into patch tokens and processes them with an encoder using self-attention to establish long-range patch dependencies.Its encoder applies normalization before MHA and FFN units and ends with an FC prediction layer.

2) Performance benefits:

Transformer optimization can improve deployment efficiency, but compression and hardware implementation remain constrained by retraining costs, weight distributions, generalization, and hardware-unfriendly operations. The section also introduces knowledge distillation as a teacher–student compression approach with task-specific and task-agnostic variants.

  • 2) Performance benefits:: Model compression can reduce latency, memory use, energy, and power, enabling more efficient transformer deployment.MobileBERT runs 5.5× faster than BERT-base on a Pixel 4 mobile phone.
  • 1) Need of Computing Resources:: Optimized models may require substantial computational resources during fine-tuning, including repeated retraining and validation iterations.The resource burden is especially associated with developing and implementing optimized models.
  • 2) Wider distribution of weights:: Transformer pruning is challenging because transformer weights have wider distributions and complex interdependencies than the compared CNN weights.These characteristics require careful treatment when removing parameters.
  • 3) Simplification prohibits generalization:: Compression can improve target-dataset performance while harming performance on datasets from different domains or with different characteristics.Removing weights trained for generalization may reduce transfer to unseen data.
  • 4) Hardware-related challenges:: Transformer attention, softmax, and multi-headed attention operations are harder to implement efficiently on specialized hardware than CNN-style linear operations.These nonlinear operations contribute to hardware-efficiency challenges.
  • Knowledge Distillation: Knowledge distillation trains a smaller student model to mimic a large teacher model using teacher predictions and soft targets.The student is trained by minimizing distillation loss alongside a task loss.
  • Knowledge Distillation: Task-specific distillation targets the same downstream application, whereas task-agnostic distillation transfers generic knowledge for multiple downstream uses.Task-specific distillation is described as suitable when optimizing performance for a particular task.

C. Methods based on distillation granularity

Distillation transfers knowledge from larger teacher transformers to smaller students at network, layer, attention, or embedding granularity. The surveyed methods combine these granularities with task-specific strategies, adaptive architectures, and pruning-related optimization, while KD can suffer limited generalization and overfitting.

  • Network-level distillation: Network-level distillation trains a student to match the teacher’s output predictions.
  • Network-level distillation: DistilBERT retains 97% of BERT’s language understanding with 40% lower model size and 60% lower latency on Intel Xeon E5-2690.
  • Task-specific and task-agnostic distillation: TinyBERT uses two-stage task-agnostic and task-specific distillation to transfer general-domain knowledge before target-dataset adaptation.
  • Layer-level distillation: Layer-level distillation matches selected teacher and student layer outputs, including hidden states from attention and feed-forward modules.
  • Adaptive distillation: DynaBERT dynamically adjusts student width and depth through two-stage KD to minimize target hardware latency, with width more robust to compression than depth.
  • Attention-based distillation: Attention-based distillation transfers teacher attention matrices to encode linguistic information such as syntax and coreference.
  • Pruning: Pruning removes redundant weights or activations, commonly by identifying low-importance parameters and iteratively fine-tuning the reduced model.
  • Pruning: oBERT reports 8.4× inference speedup with less than 1% accuracy drop and 10× speedup with less than 2% accuracy drop on Intel Xeon Platinum 8380.

C. Classification based on the matrix sparsity pattern

Transformer pruning is classified by sparsity pattern into unstructured, semi-structured, and structured methods. The surveyed techniques trade compression and accuracy against hardware efficiency, with structured patterns generally offering more deployable acceleration.

  • Unstructured pruning: Unstructured pruning removes individual parameters and can compress models strongly, but irregular sparsity often requires specialized hardware and may not yield substantial inference speedup.
  • Unstructured pruning: PLATON compresses BERT-base and ViT-B16 by up to 90% while increasing accuracy by 1.2 percentage point.
  • Semi-structured sparsity: Semi-structured sparsity is more hardware-efficient than unstructured pruning, and Nvidia A100 tensor cores accelerate the 2:4 pattern by a factor of 2.
  • Semi-structured sparsity: N×MTransformer prunes Q, K, V, attention-output, and fully connected layers while producing a model 1.7 points more accurate than SOTA N:M sparse language models.
  • Semi-structured sparsity: Sparse vision-transformer exploration methods improve DeiT-small accuracy by 0.28% while compressing at least 50% of weights.
  • Structured pruning: Structured pruning removes complete layers, filters, channels, or heads, producing regular sparse matrices that are more hardware-friendly.
  • Structured pruning: WDPruning improves DeiT-base throughput by 15% for an accuracy drop of 1% through simultaneous width and depth pruning.
  • Row/column pruning: CoFi prunes transformer operators and dimensions, achieving 10x speedup and close to 95% sparsity while preserving 90% of transformer accuracy.

3) Block Pruning:

Block and token pruning reduce transformer computation by removing redundant structures at different granularities, while adaptive methods tailor pruning to inputs. Reported results show that some approaches reduce parameters or improve accuracy, but pruning choices involve accuracy, sequence-length, and hardware tradeoffs.

  • Block pruning: Block pruning removes entire low-importance matrix blocks using block norms, achieving structured sparsity for transformer weights.HMC-Tran prunes p×q blocks whose l2-norm falls below a threshold.
  • Head pruning: 20–40% compression was achieved without quality loss by iteratively pruning redundant attention heads in Transformer and BERT models.A separate gating approach pruned half the heads with less than 0.25 BLEU loss.
  • Layer-wise pruning: Layer-wise pruning reduces transformer depth by retaining only layers with learned high impact during inference.LayerDrop learns each layer’s retention rate during training and selects a sub-network at runtime.
  • FFN pruning: Two-thirds of transformer parameters reside in FFN layers, making FFN-focused channel pruning important, especially because FFNs bottleneck CPU inference.VTP therefore emphasizes FFN channels rather than only MHA weights.
  • Quantitative comparison: Certain DeiT pruning methods reduce parameters while improving accuracy, whereas others reduce parameters with some accuracy compromise.Figures compare accuracy against parameter count for DeiT-base, DeiT-small, and DeiT-tiny.
  • Token and patch pruning: Token and patch pruning remove redundant words or image patches, and adaptive methods vary pruning according to token importance or image characteristics.Image-adaptive methods can achieve higher overall compression than fixed-rate pruning, while token thresholds can vary sequence length by input.

3) Quantitative comparison of token pruning techniques:

Token-pruning comparisons evaluate accuracy against computation while preserving the baseline model’s parameter count. The broader hardware discussion shows that realizing pruning benefits requires hardware-aware sparsity, accelerator support, and formats matched to execution patterns.

  • Token-pruning comparison: Token-pruned DeiT-small models retain the baseline parameter count while reducing multiplications, additions, and relative FLOPs.Figure 11 compares accuracy and FLOPs across token-pruning methods; circle size represents relative FLOPs.
  • Post-training pruning: Post-training pruning avoids additional retraining while maintaining baseline accuracy, unlike conventional pruning that commonly requires fine-tuning.Static methods prune once for all input lengths, whereas adaptive inference changes layer computation according to each input.
  • Hardware-aware pruning: Attention dominates BERT-large execution at sequence lengths 1024 and 2048, whereas linear layers dominate at length 256.Fan et al. classify sparsity into random, low-rank, block-wise, sliding-window, and butterfly patterns.
  • Hardware-aware pruning: ABF blocks preserve attention with butterfly-factorized linear layers, while FBF blocks replace attention with FFT to reduce parameters and computation at an accuracy cost.Combining N1 FBF and N2 ABF blocks addresses the hardware-efficiency and accuracy tradeoff.
  • Hardware-aware pruning: Weight-shape-aware pruning regularizes Q, K, V, O, and FFN matrix shapes to improve FPGA buffer and MAC utilization.The approach combines coarse-grain shape balancing with fine-grain pruning and alternating pruning and training.
  • Sparse acceleration: N:M sparse accelerators support both sparse-dense and dense-dense matrix multiplication through a unified systolic-array engine.The cited design uses nonzero-value selection for sparse-dense multiplication and reports comparable accuracy for N=1 and N=2 at fixed sparsity.
  • Sparse attention: Composite sparse attention can consume three-fourths of Longformer execution time because mixed sparse patterns have incompatible locality and storage requirements.A sparse softmax kernel processes overlapping coarse- and fine-grain patterns using BSR and CSR metadata.

I. Storage formats for sparse matrices

Sparse-matrix storage formats align pruning granularity with memory and parallelism needs, while the quantization overview organizes methods by calibration strategy, precision assignment, function, and granularity. These choices balance compression, accuracy, and hardware practicality.

  • I. Storage formats for sparse matrices: Sparse-matrix storage formats are designed to represent the different structures produced by pruning.The section introduces storage formats for sparse matrices before comparing specific schemes.
  • I. Storage formats for sparse matrices: BBP prunes rows or columns within each block, whereas BW prunes entire blocks; BBP provides higher accuracy at nearly all sparsity ratios.BBP retains more crucial information through finer-grained pruning.
  • I. Storage formats for sparse matrices: CBBWP prunes low-L2-norm blocks per column so every column retains the same number of blocks, requiring one index pointer per block.Its format supports parallelism within and across blocks and is used by an FPGA MatMul accelerator.
  • VI. QUANTIZATION: Quantization reduces parameter and activation precision to lower bit widths such as 16-bit or 8-bit integers.The overview distinguishes static versus dynamic, uniform versus mixed precision, and PTQ versus QAT.
  • 1) Quantization function:: Linear quantization maps floating-point values to equally spaced discrete levels using a scale, zero-point, range limits, and bit width.The quantization function defines Q(r) as the quantized representation of floating-point value r.
  • 1) Quantization function:: Q8BERT uses linear quantization during the forward pass and a straight-through estimator in training to learn low-precision weights and activations.The cited passage identifies Q8BERT as a quantization-aware training method.
  • 2) Matching Full Precision Model:: PTQ methods learn quantization intervals or scales by matching full-precision behavior with attention ranking, cosine similarity, correlation, or Hessian-guided objectives.Different objectives are applied separately to MHA and FFN modules in some methods.
  • 2) Matching Full Precision Model:: PSAQ-ViT performs data-free quantization by generating realistic samples from Gaussian noise properties, avoiding a calibration dataset.Its experiments reportedly outperform real-data-driven methods on benchmark models.

4) Combining pruning and quantization:

Combining pruning and quantization can reduce both computation and representation cost, but effective inference also depends on memory behavior, activation distributions, quantization granularity, and operator sensitivity. Hardware co-design further links compression choices to attainable system-level gains.

  • Combining compression methods: Joint-way compression combines pruning and quantization to obtain additional savings while avoiding two separate fine-tuning stages.The cited approach performs the compression steps jointly rather than sequentially.
  • Quantitative results: 16x speedup was achieved across language and vision tasks using 4-bit quantization and 50% weight pruning.Another cited pipeline combines 2:4 sparsity-aware fine-tuning with 8-bit post-training quantization and produces models 8x smaller.
  • Token pruning and quantization: Token pruning reduces attention and fully connected computations, with more tokens pruned from longer sentences because they contain greater redundancy.SpAtten also prunes redundant attention heads and applies progressive quantization to attention inputs.
  • Token pruning and quantization: MNNFast and A3 primarily reduce computation, whereas SpAtten also reduces memory accesses and supports both compute-bound and memory-bound models.A3 prunes QKV vectors in one head, while MNNFast prunes V vectors and therefore does not reduce FFN computation.
  • Progressive quantization: SpAtten uses more quantization bits for harder inputs and fewer bits when attention probabilities are dominated by a few tokens.The progressive scheme fetches least-significant bits only when needed and has negligible accuracy impact because softmax reduces quantization errors.
  • Hardware co-design: A top-K engine finds influential heads or tokens in O(n) time, enabling a co-designed accelerator that reduces memory access by 10.0×.The accelerator also supports splitting and concatenating most- and least-significant bits.
  • Quantization granularity: Transformer quantization can vary by layer and granularity, but differing MHA and FFN ranges make shared scale parameters vulnerable to outliers.The surveyed granularities include head-wise, channel-wise, and embedding-wise quantization.
  • Quantization granularity: Per-channel quantization improves accuracy by assigning scales independently across dimensions, but it requires more quantization parameters and inference steps.Quantizing layer normalization and softmax can degrade ViT accuracy, creating additional operator-specific constraints.

4) Per-token/Per-Embedding/Per-Patch Quantization:

Fine-grained and integer-oriented quantization reduce transformer precision and inference cost while attempting to preserve accuracy. The surveyed methods span token-, embedding-, patch-, group-, and extreme binary representations.

  • Fine-grained quantization: Per-token and per-patch quantization use distinct scale values for language-task sequences and vision patch sequences, potentially improving accuracy at the cost of additional parameters.The added quantization parameters increase with embedding size.
  • Group quantization: Group quantization partitions weights or activations into groups with distinct precision or scale, including per-embedding groups for non-uniform embedding distributions.This approach assigns different quantization parameters to different embedding groups.
  • Integer-only quantization: Integer-only quantization converts linear and nonlinear operations into a uniform integer domain, eliminating quantization and dequantization steps.I-BERT quantizes BERT layers, GELU, Softmax, and LayerNorm to Int8 and reports 2.4-4x speedup over FP32 BERT on an Nvidia T4 GPU.
  • Extreme quantization: Binarization and ternarization reduce memory requirements by 32× and 16×, respectively, but direct BERT binarization can drop GLUE accuracy by 20 points.BiBERT addresses attention quantization issues with a bi-attention module and uses BAMM for binary attention computation.
  • Accuracy and model size: Quantized models with identical weight and activation bitwidth have the same model size, while accuracy varies with knowledge transfer and outlier handling.The comparison covers PTQ-ViT, PSAQ-ViT, PSAQ-ViT V2, FQ-ViT, PTQ4ViT, and I-VIT across DeiT backbone sizes.

A. Methods for NLP

Efficient transformer designs reduce the quadratic cost of self-attention for NLP and vision models, often by approximating attention or combining attention with lightweight convolutional processing. The surveyed methods report lower complexity, latency, or resource use while retaining competitive accuracy.

  • Motivation: Self-attention updates every token using all other tokens, creating quadratic computation and costly batch-wise matrix multiplication.Efficient variants replace these operations to reduce overhead.
  • Efficient NLP attention: Reformer uses locality-sensitive hashing to reduce self-attention complexity from O(N^2) to O(NlogN) by attending among tokens within hash-based chunks.Nearby vectors are likely assigned the same hash, enabling efficient nearest-neighbor selection.
  • Efficient NLP attention: Linformer factorizes attention with linear projections, keeping latency relatively flat as sequence length increases and providing significant speedup for long inputs.Its reduced number of attention heads is compensated with long input sequences.
  • Efficient NLP attention: Performers use FAVOR+ to estimate softmax attention with linear space and time complexity, improving efficiency over quadratic Transformer and O(NlogN) Reformer attention.The method uses positive random features to approximate softmax attention.
  • Efficient NLP attention: CosFormer replaces quadratic softmax attention with a linear function using ReLU-based nonnegative features and cosine reweighting for local correlations.Its linear formulation changes the multiplication order by computing Key–Value products before attention–Key products.
  • Efficient vision attention: MobileViTV2 uses separable self-attention with element-wise operations, reducing complexity to O(N) and achieving 75.6% ImageNet accuracy with 3.2× faster iPhone12 inference than MobileViT.It performs 1% better than MobileViT despite having more parameters.
  • Efficient vision architectures: EdgeNeXt combines depthwise convolution, pointwise operations, and channel-wise cross-covariance attention to reduce attention complexity from quadratic to linear.It achieves 71.2% ImageNet accuracy with 1.3M parameters and is 1.64 times faster than EfficientNet on TPUV3.

C. Quantitative comparison of lightweight CV transformer techniques

Lightweight vision transformer comparisons show a generally positive but non-linear relationship between parameter count and ImageNet accuracy. Architecture and training choices can produce substantial accuracy differences at similar model sizes, with diminishing returns from further scaling.

  • Comparison setup: ImageNet top-1 accuracy is compared against parameter count for lightweight vision transformers, baseline transformers, and CNN models.The compared vision transformers include MobileViT, MobileViTV2, Mobile-former, LVT, TFormer, and EdgeNeXt.
  • Observed trade-off: Larger models generally achieve higher accuracy, but the relationship between parameter count and accuracy is positive rather than necessarily linear.The comparison includes DeiT-tiny, ViT-small, T2T-ViT, and MobileNet baselines.
  • Observed trade-off: Models with similar parameter counts can have widely different accuracy scores, indicating that architecture and training process also matter.The passage identifies these factors as important in determining model performance.
  • Observed trade-off: Adding parameters beyond a certain point may not significantly improve accuracy, while MobileViT achieves high accuracy with a relatively small model size.T2T-ViT has more parameters than the compared models but lower accuracy than MobileViTV2.
  • NAS context: Transformer NAS automates architecture design, and hardware-aware NAS searches for models balancing accuracy with compute performance on target hardware.The surveyed section classifies transformer NAS methods and discusses their search spaces, strategies, and evaluation phases.
  • Hardware-aware NAS: Hardware-aware searches can specialize models to platforms: HAT reports that GPU-specialized NLP transformers run faster on GPU than CPU, with the reverse for CPU-specialized models at similar validation accuracy.The hardware metrics are included as part of a multi-objective optimization function.

2) Hybrid Attention-Convolution Search Space:

Hybrid attention-convolution search spaces combine transformer attention parameters with convolutional choices, enabling NAS to explore architectures suited to different tasks and hardware constraints. The surveyed methods use reinforcement learning, one-shot, evolutionary, and related strategies to find efficient models.

  • Search space: Hybrid search spaces include attention and convolution parameters, such as attention heads, FFN dimensions, convolution kernel sizes, and channel sizes.Examples include TextNAS, GLiT, and BurgerFormer.
  • Search strategies: NAS search strategies include reinforcement learning, one-shot or differentiable search, evolutionary learning, once-for-all search, random search, proxy search, and Bayesian optimization.These strategies are used to select architectures from predefined primitive operations.
  • One-shot search: One-shot methods train a weight-sharing supernetwork so multiple architecture combinations can be evaluated without separately training each candidate.DARTS assigns learnable architectural parameters to operations in the search space.
  • Differentiable search: Planer searches under a target latency value and produces a sparsely activated network with 2× GPU speedup while maintaining baseline accuracy.Its search space includes FFN layers, attention heads, and mixture-of-expert layers.
  • Evolutionary search: Evolutionary methods use selection and mutation to iteratively improve architectures according to a fitness function.Evolved Transformer and Primer apply evolutionary learning to encoder-decoder and decoder-only attention networks, respectively.
  • Hardware-aware search: A hardware-aware style-transfer search found networks at least 2.1× faster than the baseline on Xiaomi Redmi 10 and Raspberry Pi 3 devices.The ViT backbone submodules were searched using evolutionary search.
  • Once-for-all search: Once-for-all search trains a maximum-dimension supernetwork, then samples and validates subnetworks without additional fine-tuning.HAT and several vision-transformer methods use this approach to specialize models for metrics such as accuracy, model size, and latency.
  • Distillation-assisted search: Knowledge distillation can accelerate NAS by transferring knowledge from a large teacher network while evaluating subnetworks during the search.LightHuBERT trains a once-for-all BERT supernetwork with a pre-training distillation loss.

D. Application of NAS for Model Compression

NAS applies automated search to transformer compression, mixed-precision quantization, hybrid operators, and hardware-aware model design. The surveyed results connect these searches with accuracy, parameter count, latency, and platform-specific efficiency.

  • NAS-based pruning: NAS-based pruning automatically removes redundant parameters to compress transformer models for downstream tasks.NAS-BERT is task-independent, while AdaBERT is task-dependent and uses differentiable search.
  • Mixed-precision quantization: Mixed-precision quantization searches layer-specific bit-width assignments because an L-layer transformer has b^L possible configurations.AQ-BERT searches different precisions across encoder layers and transformer subcomponents.
  • Hybrid operators: NAS can select conventional O(n^2) or linear O(n) attention at each transformer layer to balance computational cost and accuracy.The search space includes both attention types and chooses their placement across layers.
  • Hybrid operators: ShiftAddNAS searches multiplication and non-multiplication operators across backbone layers to trade accuracy against hardware efficiency.Addition- and bitwise-shift-only networks reduce expensive multiplications but otherwise attain inferior accuracy.
  • Empirical tradeoffs: NASformer achieves better accuracy with fewer parameters than manually designed Swin and DeiT models on ImageNet.The surveyed plot shows accuracy increasing with parameter count before eventually saturating.

C. Skipping redundant, ineffectual or trivial computations

The surveyed techniques reduce transformer computation by predicting which attention interactions matter, exploiting low precision, and bypassing redundant or ineffectual operations. These methods target attention scoring, softmax, and value aggregation while sometimes trading exactness for efficiency.

  • Motivation: Attention computation can skip redundant work because many operations are repeated, ineffectual, or trivial.Examples include patch-locality repetition, low-impact attention weights, and multiplication by zero or one.
  • Approximate attention: Low-precision Q and K preserve attention-score ordering because quantization and exponentiation are monotonic, reducing 4-bit attention overhead to 1/16 of 16-bit dense attention.The approach relies on softmax depending on relative rather than absolute score values.
  • Approximate attention: ELSA uses hashed key-query similarity to identify keys likely to receive high attention scores before performing full attention computation.It estimates vector angles with Hamming distances between structured hashes.
  • Attention pruning: DOTA predicts weak attention connections from low-rank-transformed queries and keys, then masks them after optimizing network and detection losses jointly.The method addresses the softmax-quality change caused by masking weak attentions.
  • Attention pruning: A3 preprocesses keys and greedily estimates row rough-scores, processing only rows with positive rough-scores during query response.The key matrix is sorted during knowledge comprehension to reduce critical-path delay.
  • Hardware acceleration: Approximate attention accelerators exploit near-zero softmax weights, adaptive precision, and sparsity prediction to avoid redundant computations.Reported designs include Ham et al.'s thresholded-row processing, Chen et al.'s ternary key representation, and Wang et al.'s zero-product scheduling.

D. Dataflows for exploiting reuse

Transformer accelerators use dataflows and data reorganization to exploit sharing in inputs, weights, outputs, and sparse attention structures. The surveyed designs reduce memory movement or improve utilization by matching computation layouts to transformer reuse patterns.

  • Stationary dataflows: Input-, weight-, and output-stationary dataflows reduce energy by reusing inputs, weights, or partial sums, respectively.The best choice depends on the dominant reuse pattern of the accelerator workload.
  • Stationary dataflows: Output-block stationary dataflow broadcasts blocks and then rows or columns within blocks, lowering memory accesses and output-writing bandwidth versus output-stationary dataflow.Its reported hardware utilization is 89%, compared with 96% for the comparison design.
  • Sparse attention dataflows: Score-stationary dataflow stores sparse attention scores in processing elements and inserts ghost PEs to alleviate sparse-decoding overhead.The design targets non-uniform accesses caused by sparse score matrices feeding sparse matrix multiplication.
  • Sparse attention dataflows: SALO uses diagonal connections and global row/column processing to reuse vectors in sliding-window, dilated-window, and global attention.Its datapath targets reuse between neighboring queries and between local and global attention.
  • Sparse attention dataflows: SALO's five-stage attention pipeline combines output-stationary QK multiplication, approximate exponentiation, row-wise accumulation, normalization, and weight-stationary value aggregation.The architecture reuses K/V vectors for global tokens and Q vectors across global attention processing.
  • Vision-transformer dataflows: ViTCoD prunes more than 90% of ViT attention maps with fixed layouts, clusters query-key pairs into denser and sparser patterns, and uses separate engines for each layout.The design exploits fixed ViT token counts and reduces irregularity in sparse maps.

E. Block-circulant matrix for reducing weight storage

Block-circulant matrices reduce transformer weight storage by representing circulant blocks with compact vectors and replacing matrix-vector multiplication with FFT operations. Ftrans extends this representation while placing major layers on chip and balancing pipeline resources.

  • Block-circulant representation: Block-circulant matrices store one index vector per circulant block instead of the full weight matrix.Each row or column is a cyclic reformulation of the others.
  • Block-circulant representation: FFT-based multiplication reduces circulant matrix-vector complexity from O(b^2) to O(b log b), where b is the row or column size.The complexity reduction accompanies the compact circulant representation.
  • Ftrans: Ftrans improves block-circulant representation by encoding the remaining rows and columns rather than using only the first row or column.Its representation is intended to capture parameter values more accurately while maintaining accuracy.
  • Ftrans: Ftrans stores encoder and decoder layers on chip, while placing the embedding layer off chip because the former account for two-thirds of total parameters.It uses dedicated processing elements for matrix-vector multiplication and FFT/IFFT operations.
  • Results: Ftrans reports up to 16X model-size reduction while also improving performance and energy efficiency.Resource scheduling assigns more processing elements to slower layers to balance the pipeline.
  • Future directions: The survey identifies scalability, interpretability, fair comparison, and reliability as continuing challenges for transformer-inference optimization research.It also highlights needs for AutoML compression, hardware-aware CV-transformer NAS, NAS benchmarks, and transformer inference benchmarks.
Loading 2307.07982v1…