PaperlayerView exact revision on arXiv

Source-linked AI summary

ViTCoD: Vision Transformer Acceleration via Dedicated Algorithm and Accelerator Co-Design

Haoran You, Zhanyi Sun, Huihong Shi, Zhongzhi Yu, Yang Zhao, Yongan Zhang, Chaojian Li, Baopu Li, Yingyan Celine Lin

arXiv:2210.09573v3cs.LGcs.ARcs.CV

TL;DR

Vision Transformer self-attention is a major efficiency bottleneck, while NLP-oriented sparse accelerators are poorly matched to ViTs’ fixed, highly sparse attention patterns. ViTCoD co-designs polarized fixed-pattern attention, Q/K auto-encoding, and dedicated hardware, achieving large speedups over general platforms and prior accelerators while maintaining accuracy.

  • Problem

    ViT self-attention is a major inference-efficiency bottleneck, and NLP-oriented accelerators are not optimal because ViTs use highly sparse fixed patterns while NLP Transformers use dynamic patterns.

  • Method

    ViTCoD prunes and polarizes attention maps into denser or sparser fixed patterns, compresses Q/K vectors with a learnable auto-encoder, and uses dedicated hardware for both workloads and encoder/decoder engines.

  • Results

    235.3×, 142.9×, 86.0×, 10.1×, and 6.8× attention speedups were achieved over CPU, EdgeGPU, GPU, SpAtten, and Sanger at 90% sparsity while maintaining accuracy.

  • Takeaways & Limitations

    ViTCoD demonstrates that jointly exploiting fixed sparse attention and reducing Q/K data movement can improve sparse ViT acceleration efficiency.

Abstract

from arXiv · show

Vision Transformers (ViTs) have achieved state-of-the-art performance on various vision tasks. However, ViTs' self-attention module is still arguably a major bottleneck, limiting their achievable hardware efficiency. Meanwhile, existing accelerators dedicated to NLP Transformers are not optimal for ViTs. This is because there is a large difference between ViTs and NLP Transformers: ViTs have a relatively fixed number of input tokens, whose attention maps can be pruned by up to 90% even with fixed sparse patterns; while NLP Transformers need to handle input sequences of varying numbers of tokens and rely on on-the-fly predictions of dynamic sparse attention patterns for each input to achieve a decent sparsity (e.g., >=50%). To this end, we propose a dedicated algorithm and accelerator co-design framework dubbed ViTCoD for accelerating ViTs. Specifically, on the algorithm level, ViTCoD prunes and polarizes the attention maps to have either denser or sparser fixed patterns for regularizing two levels of workloads without hurting the accuracy, largely reducing the attention computations while leaving room for alleviating the remaining dominant data movements; on top of that, we further integrate a lightweight and learnable auto-encoder module to enable trading the dominant high-cost data movements for lower-cost computations. On the hardware level, we develop a dedicated accelerator to simultaneously coordinate the enforced denser/sparser workloads and encoder/decoder engines for boosted hardware utilization. Extensive experiments and ablation studies validate that ViTCoD largely reduces the dominant data movement costs, achieving speedups of up to 235.3x, 142.9x, 86.0x, 10.1x, and 6.8x over general computing platforms CPUs, EdgeGPUs, GPUs, and prior-art Transformer accelerators SpAtten and Sanger under an attention sparsity of 90%, respectively.

I. INTRODUCTION

ViTCoD targets ViT self-attention, a major latency bottleneck that differs from NLP Transformers because ViTs support highly sparse fixed patterns. It combines polarized attention workloads, auto-encoder compression, and dedicated hardware to improve efficiency while maintaining accuracy.

  • I. INTRODUCTION: ViT self-attention accounts for over 50% of total latency on EdgeGPU platforms, making its quadratic attention computation a major inference bottleneck.The bottleneck is associated especially with core matrix multiplications among Q/K/V vectors.
  • I. INTRODUCTION: ViTs can reach 90%–95% sparsity with fixed patterns, unlike NLP Transformers, which typically require dynamic patterns for only 50%–70% sparsity.This difference motivates designs specialized for fixed ViT workloads rather than highly reconfigurable NLP-oriented accelerators.
  • I. INTRODUCTION: ViTCoD prunes and polarizes attention maps into denser and sparser fixed patterns, regularizing two workload levels while reducing attention computations without severely hurting accuracy.The framework is presented as the first co-design framework dedicated to sparse ViT inference.
  • I. INTRODUCTION: ViTCoD adds a lightweight learnable auto-encoder and dedicated two-pronged hardware with encoder/decoder engines to trade costly data movement for lower-cost computation.The hardware coordinates regular denser workloads and reduced irregular sparser workloads while compressing Q/K vectors before off-chip transfers.
  • I. INTRODUCTION: 235.3×, 142.9×, 86.0×, 10.1×, and 6.8× speedups were achieved over CPUs, EdgeGPUs, GPUs, SpAtten, and Sanger at 90% attention sparsity while maintaining model accuracy.These results summarize the reported comparison across general platforms and prior Transformer accelerators.

II. RELATED WORKS

Prior work includes sparse attention algorithms, sparse tensor accelerators, and Transformer-specific hardware. ViTCoD differs by exploiting ViTs’ fixed sparse patterns rather than dynamically predicting NLP-style masks.

  • II. RELATED WORKS: Sparse attention algorithms address Transformers’ quadratic sequence-length complexity using mechanisms including random, window, global, locality-sensitive, or fixed sparse patterns.The related work spans both NLP Transformers and ViTs.
  • II. RELATED WORKS: ViTCoD is distinguished as the first algorithm–accelerator co-design framework dedicated to sparse ViTs, exploiting their fixed sparse patterns from both algorithmic and hardware perspectives.Earlier NLP-focused approaches require dynamic, input-dependent sparse-mask prediction.
  • II. RELATED WORKS: Sparse tensor accelerators explore specialized dataflows, sparse storage, data reuse, and memory-bandwidth improvements because sparse matrix operations are inefficient on general-purpose CPUs and GPUs.These designs provide hardware precedents for handling irregular sparse computation.
  • II. RELATED WORKS: Existing Transformer accelerators use approximation, structural pruning, low precision, mask prediction, packing, reconfigurable architectures, or locality-aware execution, mainly for NLP workloads.Several approaches incur accuracy drops or achieve limited sparsity, while their dynamic masks require flexible hardware.
  • II. RELATED WORKS: Self-attention accounts for over 50% of measured ViT latency on an EdgeGPU TX2, motivating dedicated acceleration alongside the related sparse-attention approaches.The cited breakdown contrasts FLOPs and measured latency across ViT models.

III. VITCOD: MOTIVATION & OVERVIEW

ViTCoD targets self-attention as a major ViT inference bottleneck by combining sparse attention, compact representations, and dedicated hardware to reduce computation and data movement.

  • Over 50% of ViT inference latency is attributed to self-attention on commercial edge platforms, reaching 53% on EdgeGPUs.
  • ViTCoD prunes attention maps by up to 90% sparsity, polarizes them into denser or sparser fixed patterns, and uses an auto-encoder to compact attention vectors.The framework is presented as a response to irregular accesses, workload imbalance, and data-movement costs in sparse ViT attention.
  • Self-attention computes query-key correlations, normalizes them with Softmax, and forms weighted sums of value embeddings across multiple heads.
  • ViTCoD’s overview combines the sparse-attention algorithm with a dedicated accelerator designed to improve utilization and reduce data movement.
  • Sparse attention changes the two core multiplications into sampled dense-dense and sparse-dense operations, producing irregular accesses to Q, K, and V.

B. ViTCoD’s Split and Conquer Algorithm

ViTCoD’s split-and-conquer algorithm creates fixed sparse attention masks and reorganizes them into two workload patterns, improving regularity for hardware acceleration.

  • B. ViTCoD’s Split and Conquer Algorithm: ViTCoD prunes attention using fixed masks and reorders the remaining entries to enforce only denser or sparser computation workloads.The design targets the costly quadratic dependence of self-attention on the number of tokens or patches.
  • B. ViTCoD’s Split and Conquer Algorithm: The algorithm uses the relatively fixed number of ViT input patches to avoid the dynamic pattern prediction required by variable-length NLP Transformers.
  • B. ViTCoD’s Split and Conquer Algorithm: For each query, pruning retains high-value attentions until their normalized cumulative score reaches threshold θp, producing a binary mask.The mask marks retained attentions with 1 and pruned attentions with 0.
  • B. ViTCoD’s Split and Conquer Algorithm: Attention-map reordering clusters query-key pairs into a denser global-token pattern and a sparser pattern containing the remaining mostly zero entries.Adjacent patches tend to correlate more strongly than distant patches, while global tokens correlate broadly.
  • B. ViTCoD’s Split and Conquer Algorithm: Visualization across 144 DeiT-Base attention heads shows a clustered dense block alongside highly sparse remaining attentions, improving regularity.The dense block can use a dense computing engine, while the sparse remainder is handled separately.

C. ViTCoD Learnable Auto-encoder Module

ViTCoD’s learnable auto-encoder compresses Q/K vectors before costly off-chip transfers and reconstructs them for attention, addressing data-movement bottlenecks while maintaining accuracy.

  • C. ViTCoD Learnable Auto-encoder Module: Reducing Q/K dimensions can introduce low-rank approximation that degrades achievable accuracy, motivating reconstruction-based compression instead.The cited limitation is associated with shrinking Q/K dimensions directly.
  • C. ViTCoD Learnable Auto-encoder Module: Compressing Q/K vectors before off-chip transfer and reconstructing them afterward reduces costly data movements while maintaining model accuracy.The module exploits redundancy across attention heads and is jointly optimized with the ViT model weights.
  • C. ViTCoD Learnable Auto-encoder Module: The auto-encoder is lightweight and learnable, using reconstruction loss to keep recovered Q′/K′ close to the original vectors.The training objective combines cross-entropy and reconstruction losses.
  • C. ViTCoD Learnable Auto-encoder Module: Finetuning significantly reduces both test and reconstruction losses, and fully recovers accuracy after inserting the auto-encoder into a pretrained ViT.These results support convergence and effectiveness of the module.

D. The Unified ViTCoD Algorithm

ViTCoD combines split-and-conquer attention polarization with auto-encoding: the former regularizes sparse workloads, while the latter trades costly data movement for computation.

  • D. The Unified ViTCoD Algorithm: The unified ViTCoD algorithm combines workload polarization with auto-encoding to reduce attention computations and mitigate data-movement costs.Split and conquer produces two attention patterns, while the auto-encoder compresses representations to improve utilization.
  • D. The Unified ViTCoD Algorithm: Split and conquer reduces attention computations and polarizes imbalanced workloads into two patterns, while auto-encoding addresses the resulting utilization problem.The two components explore complementary directions for efficient ViT inference.

A. Motivation of ViTCoD Accelerator

ViTCoD’s accelerator motivation arises from ViTs’ fixed sparse patterns and high sparsity, which create opportunities for regular designs but also severe irregularity, imbalance, and data-movement challenges.

  • A. Motivation of ViTCoD Accelerator: Fixed and structurally sparse attention allows ViTCoD to avoid on-the-fly mask prediction and highly reconfigurable processing elements used for NLP Transformers.ViT-specific fixed patterns create an opportunity for dedicated acceleration.
  • A. Motivation of ViTCoD Accelerator: ViTCoD compresses Q/K representations to 50% of their original size, trading modest extra reconstruction computation for reduced data movement.High sparsity and diagonal patterns limit vector reuse, making data movement a bottleneck.
  • A. Motivation of ViTCoD Accelerator: High sparsity reduces computation but aggravates irregular accesses, workload imbalance, and processing-element under-utilization.These challenges motivate specialized micro-architecture and dataflow choices.
  • A. Motivation of ViTCoD Accelerator: S-stationary dataflow reuses K vectors with a small on-chip buffer, whereas K-stationary processing suits sparse attention by computing only paired Q/K vectors.The alternatives expose different reuse and sparsity advantages.

B. ViTCoD Accelerator’s Micro-architecture

The ViTCoD accelerator uses separate denser and sparser engines, dynamically allocates processing elements, and applies specialized tiling, indexing, buffering, and dataflows to improve utilization.

  • B. ViTCoD Accelerator’s Micro-architecture: Two separate computing engines process denser and sparser workloads, reducing off-chip accesses and improving PE/MAC utilization.The architecture also includes memory hierarchy and dedicated PE-array organization.
  • B. ViTCoD Accelerator’s Micro-architecture: Dynamic PE allocation assigns resources between denser and sparser engines in proportion to their known workload sizes.Fixed masks allow workload estimation before execution.
  • B. ViTCoD Accelerator’s Micro-architecture: Fine-grained tiling maps S/V vectors spatially and accumulates feature-dimension partial sums temporally, reusing S and V with a small on-chip buffer.PE lines are reconfigured between inter-PE and intra-PE accumulation across phases.
  • B. ViTCoD Accelerator’s Micro-architecture: The sparser engine uses CSC indexing and query-based Q forwarding to process attention maps whose density is below 10%.Preloaded nonzero indexes support sparse execution with reduced data and computation density.
  • B. ViTCoD Accelerator’s Micro-architecture: Dedicated buffers, sparse/dense multiplication control, SoftMax units, and activation units support both workload types within the accelerator.The functional units provide local reuse and separate handling of dense and indexed sparse operations.

2) Encoder and Decoder Engines:

ViTCoD equips its accelerator with dedicated encoder and decoder engines, compiler-based task reconfiguration, and hardware resources tailored to sparse ViT workloads.

  • Encoder and Decoder Engines: Dedicated encoder and decoder engines process auto-encoder workloads, with small weights pre-loaded on chip.The engines use their own PE/MAC lines, while auto-encoder weights occupy small on-chip storage.
  • Reconfigurable Hardware: One-time compilation adapts the accelerator to different ViT mask patterns or head numbers by generating task-specific processing instructions.A parser extracts configurations such as global-token counts, buffer sizes, and dataflows before compilation.
  • Evaluation Setup: Experiments cover DeiT, LeViT, and Strided Transformer models on ImageNet and Human3.6M, against CPU, EdgeGPU, GPU, SpAtten, and Sanger baselines.The benchmark includes both general computing platforms and dedicated attention accelerators.
  • Hardware Platform Setup: The evaluated setup uses 3 mm2 area, 76.8GB/s DDR4 bandwidth, 323.9mW power at 500MHz, 320KB SRAM, and 512 MACs.Performance is evaluated with a cycle-accurate simulator using post-layout-derived MAC and memory-access costs.
  • Evaluation Setup: ViTCoD’s accelerator floorplan and normalized speedups are reported through dedicated layout and performance figures.The supplied figures identify the accelerator floorplan and speedup comparison across seven ViT models.

B. Overall Performance Comparison

ViTCoD substantially reduces ViT attention latency while preserving accuracy, and its auto-encoder recovers most compression-induced accuracy loss after finetuning.

  • Accuracy–Latency Trade-offs: 45.1%–85.8% and 72.0%–84.3% attention-layer latency reductions are achieved for DeiT and LeViT, respectively, with less than 1% accuracy drop.Across 50%–95% sparsity, DeiT and LeViT maintain 90% and 80% sparsity, respectively, with negligible accuracy loss.
  • Auto-Encoder Evaluation: Training trajectories evaluate auto-encoder-equipped LeViTs against vanilla LeViT accuracy references.Dashed lines denote the vanilla models’ accuracy.
  • Auto-Encoder Evaluation: Finetuning recovers auto-encoder-augmented ViTs to within 0.5% accuracy of the corresponding models.The result supports compressing Q/K vectors to reduce costly data movement with negligible overhead.
  • Ablation Studies: Pruning strengthens attention-map polarization, providing 5.14× average speedup across 60%–90% pruning and 8.14× at 90% sparsity.The comparison isolates pruning from reordering and attributes the stronger regularity to sparser sparse parts.

D. Evaluation of the ViTCoD Accelerator

Across sparse ViT benchmarks, ViTCoD improves attention speed and energy efficiency over general platforms and prior accelerators by exploiting regular sparse patterns and compact Q/K representations.

  • Overall Performance: 235.3×, 142.9×, 86.0×, 10.1×, and 6.8× attention speedups over CPU, EdgeGPU, GPU, SpAtten, and Sanger are achieved at 90% sparsity.Across 60%–90% sparsity, average speedups remain 127.2×, 77.0×, 46.5×, 6.8×, and 4.3× over the same baselines.
  • Overall Performance: ViTCoD improves energy efficiency by 9.8× over Sanger while evaluating attention layers across six DeiT and LeViT models.The evaluation compares normalized efficiency and latency speedups against five baselines.
  • Latency Breakdown Analysis: The split-and-conquer algorithm provides 2.7× speedup over Sanger, and adding the auto-encoder provides a further 2.5×.The breakdown separates algorithmic and auto-encoder contributions to latency.
  • Latency Breakdown Analysis: Auto-encoder integration reduces ViTCoD’s data-movement share from 50% to 28%, addressing a key performance bottleneck.Sanger reduces movement through Q/K reuse but incurs larger computation workloads, whereas ViTCoD trades movement for computation.
  • Design Rationale: ViTCoD combines fixed polarized sparse attention with compact Q/K representations and dedicated hardware support for denser and sparser workloads.The framework also includes encoder and decoder engines to cooperate with auto-encoder modules.
Loading 2210.09573v3…