PaperlayerView exact revision on arXiv

Source-linked AI summary

A$^3$: Accelerating Attention Mechanisms in Neural Networks with Approximation

Tae Jun Ham, Sung Jun Jung, Seonghak Kim, Young H. Oh, Yeonhong Park, Yoonho Song, Jung-Hun Park, Sanghee Lee, Kyoung Park, Jae W. Lee, Deog-Kyoon Jeong

arXiv:2002.10941v1cs.DCcs.LG

TL;DR

Attention is often implemented as dense matrix-vector multiplication even though it is a search in which much computation has little impact on the output. A3 combines approximation-based candidate selection with specialized parallel hardware, achieving multiple orders of magnitude speedup and energy-efficiency gains over conventional hardware.

  • Problem

    Dense matrix-vector implementations perform many computations with little impact on the attention output, despite attention being a search operation.

    Source: C. Opportunity for Approximation · p. 3

  • Method

    A3 reduces attention computation through approximate candidate selection and a specialized hardware pipeline that exploits parallelism.

    Source: §I · pp. 1–2

  • Results

    Multiple orders of magnitude speedup and energy-efficiency improvements over conventional hardware are reported for A3.

    Source: §I · p. 2

  • Takeaways & Limitations

    A3 could enable existing attention-based neural networks to use larger external knowledge bases or longer sequences.

    Source: §I · p. 2

Abstract

from arXiv · show

With the increasing computational demands of neural networks, many hardware accelerators for the neural networks have been proposed. Such existing neural network accelerators often focus on popular neural network types such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs); however, not much attention has been paid to attention mechanisms, an emerging neural network primitive that enables neural networks to retrieve most relevant information from a knowledge-base, external memory, or past states. The attention mechanism is widely adopted by many state-of-the-art neural networks for computer vision, natural language processing, and machine translation, and accounts for a large portion of total execution time. We observe today's practice of implementing this mechanism using matrix-vector multiplication is suboptimal as the attention mechanism is semantically a content-based search where a large portion of computations ends up not being used. Based on this observation, we design and architect A3, which accelerates attention mechanisms in neural networks with algorithmic approximation and hardware specialization. Our proposed accelerator achieves multiple orders of magnitude improvement in energy efficiency (performance/watt) as well as substantial speedup over the state-of-the-art conventional hardware.

I. INTRODUCTION

A3 addresses the high computational cost of attention with approximate candidate selection and specialized hardware. The paper claims multiple orders of magnitude improvements in speedup and energy efficiency over conventional hardware, with minimal accuracy degradation from approximation.

I. INTRODUCTION

  • Prior FPGA and ASIC accelerators improve CNN and RNN processing, but do not fully support emerging attention mechanisms, despite neural networks’ high computational cost.The introduction says neural networks can involve tens of billions of floating-point operations.Source: §I · p. 1
  • Attention is a content-based similarity search that lets neural networks retrieve relevant information from past states or external memory, and is used across computer vision and natural language processing.The introduction also notes its use for complex problems that conventional neural networks struggle with because they cannot memorize and retrieve data.Source: §I · p. 1
  • Attention computation grows with the number of search targets and can grow quadratically for self-attention, making it a significant performance and energy cost that limits model capacity.More targets arise from larger knowledge bases, longer histories, or longer input sequences.Source: §I · p. 1
  • A3 combines approximate candidate selection that avoids exhaustive search with a specialized parallel hardware pipeline to accelerate attention and improve energy efficiency.The selection scheme reduces the number of search targets because not all targets are equally likely to be relevant.Source: §I · pp. 1–2
  • A3 achieves multiple orders of magnitude speedup and energy-efficiency improvements over conventional hardware, with further gains from specialized approximation hardware and minimal model-accuracy degradation.The paper describes energy efficiency as performance per watt.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION · A. Attention Mechanism · B. Cost of Attention Mechanism

Attention is a differentiable content-based search that scores keys against a query, normalizes the scores, and uses them to retrieve a weighted sum of values. Its computation grows with the number and dimension of entries, and profiling finds it accounts for a substantial share of inference and query-response time.

A. Attention Mechanism

  • Attention computes query-key dot products, applies softmax, and uses the normalized scores to retrieve a weighted sum from the value matrix.Because its computations are differentiable, soft attention is well suited to neural networks trained with back-propagation.Source: A. Attention Mechanism · p. 2
  • In bAbI QA, attention can identify “John moved to the garden.” as relevant to “Where is John?” and repeat retrieval when multiple sentences are needed.The End-to-End Memory Network embeds statements and the query, retrieves relevant sentences, then uses a final weight matrix to generate the answer.Source: A. Attention Mechanism · p. 2, Figure 2 · p. 2

B. Cost of Attention Mechanism

  • For n entries of dimension d, attention uses nd multiplications in each of Steps 1 and 3, with the operation counts increasing almost linearly in n and d.Step 1 also uses n(d − 1) additions; Step 2 uses n exponent computations, n − 1 additions, and n divisions; Step 3 uses (n − 1)d additions.Source: B. Cost of Attention Mechanism · pp. 2–3
  • Here, n is the number of knowledge-base entries or past states searched, while d is the dimension of each vector embedding.Larger n lets models search more data and is more likely than d to increase further; larger d can provide richer, higher-quality embeddings.Source: B. Cost of Attention Mechanism · p. 3
  • The profiling covers MemN2N on bAbI QA, KV-MemN2N on WikiMovies QA, and BERT on SQuAD, using an Intel Xeon Gold 6128 CPU and, for BERT, an NVIDIA Titan V GPU.The workloads and platform scope the attention-runtime analysis.Source: B. Cost of Attention Mechanism · p. 3
  • Attention accounts for over 35% of inference runtime in all profiled workloads and over 70% of query-response time in MemN2N and KV-MemN2N.BERT’s attention share remains the same for inference and query response because it performs comprehension and query response in an integrated manner.Source: B. Cost of Attention Mechanism · p. 3, Figure 3 · p. 3
  • To accelerate attention and improve its energy efficiency, the paper proposes a specialized hardware accelerator for the operation.The paper presents its design in Section III.Source: B. Cost of Attention Mechanism · p. 3

C. Opportunity for Approximation · III. A3 BASE DESIGN

A3 targets wasted computation in dense attention by identifying likely high-scoring key rows before computing attention outputs. Its base design is a specialized pipelined accelerator, while the design described here is without approximation.

C. Opportunity for Approximation

  • Most key-query dot-product scores have little effect on the output because softmax makes most scores near-zero.The passage characterizes softmax as a continuous, differentiable approximation of argmax.Source: C. Opportunity for Approximation · p. 3
  • Popular implementations compute attention with key-query matrix-vector multiplication, softmax, and value-weight matrix-vector multiplication.Frameworks such as TensorFlow and Torch provide fast matrix-vector multiplication; the dense implementation is functionally correct.Source: C. Opportunity for Approximation · p. 3
  • A3 preprocesses the key matrix to identify likely high-scoring candidate rows, avoiding unnecessary dot-product, softmax, and final-result computation.The proposed approximation scheme is presented in Section IV; Section V covers the hardware module for it.Source: C. Opportunity for Approximation · p. 3

III. A3 BASE DESIGN

  • A3 is a specialized attention accelerator designed for integration with a CPU, GPU, or existing hardware accelerator.The base design accelerates the attention mechanism shown in Figure 1 of Section II.Source: §III · p. 3
  • A3 uses a pipelined customized datapath that exploits parallelism to target high throughput and energy efficiency.Source: §III · p. 3
  • The base design presented in this section operates without approximation; approximate attention and extended hardware designs are introduced later.Source: §III · p. 3

A. Pipeline Design

Base A3 maps key and value matrices plus a query vector to a d-dimensional output through dot-product, exponent-computation, and output-computation modules.

A. Pipeline Design

  • The dot-product module uses d multipliers and a d-way adder tree for each key row, stores the n results, and finds their maximum.The maximum is used by the exponent-computation module.Source: A. Pipeline Design · p. 4
  • Subtracting the maximum dot product before exponentiation keeps inputs nonpositive and exponent outputs at most 1; splitting inputs into parts reduces lookup-table size.For 8-bit input, the decomposition uses two 4-bit exponent lookups and a multiplier; the example tables each have 256 entries instead of one 65,536-entry table.Source: A. Pipeline Design · p. 4
  • The output module divides each score by the accumulated score sum, uses the normalized weight to scale its corresponding value row, and accumulates the result.Base A3's three modules implement a reordered computation pattern suited to hardware.Source: A. Pipeline Design · p. 4
  • The pipeline handles three queries at a time, with latency of 3n + 27 cycles and throughput of n + 9 cycles per query.The modules are designed to match throughput, each taking n cycles + α per query.Source: A. Pipeline Design · pp. 4–5

B. Quantization · C. Design Details

A3 uses stage-specific fixed-point widths to preserve precision, prevent overflow, and reduce energy. Its design supports preloaded key/value matrices, queued query processing, parallel execution, and configurable workload dimensions.

B. Quantization

  • A3 quantizes floating-point inputs to i integer and f fraction bits plus a sign bit, then assigns stage-specific widths to preserve precision, prevent overflow, and reduce energy.The paper notes that register and computation-unit energy scales linearly or quadratically with bitwidth, and that fixed-point operations cost less energy than floating-point operations.Source: B. Quantization · p. 5
  • Integer widths follow dynamic range: temp uses 2i bits, dot_product uses log2(d)+2i bits plus an extra integer bit for exponent subtraction, expsum uses log2(n) bits, and output uses i+log2(n) bits.score[] and weight[] need no integer bits because their values are bounded between 0 and 1.Source: B. Quantization · p. 5
  • Fraction widths are 2f for products, dot_product, scores, and weights, while the final output uses 3f bits.The 3f output comprises 2f bits for weights and f bits for values; the evaluation uses f = 4.Source: B. Quantization · p. 5

C. Design Details

  • Key and value matrices are copied to A3’s SRAM beforehand, while query transfer is included in response time and queries and outputs pass through queues.A query starts when a dot-product module becomes available; the passage gives question answering as an example where key/value transfer precedes query response time.Source: C. Design Details · p. 5
  • Independent attention computations can use separate A3 units, while queries sharing key/value matrices can be pipelined on one unit or spread across units for higher throughput.Source: C. Design Details · p. 5
  • A3 is evaluated at n = 320 and d = 64; larger n can place vectors beyond the first n in DRAM and use prefetching to hide memory latency.The key/value matrices for a BERT model with n = 320 and d = 64 are described as small enough for SRAM, making DRAM unlikely to be needed in the near future.Source: C. Design Details · pp. 5–6
  • Because d is unlikely to vary widely, A3 can use a sufficiently large d and zero-pad smaller choices; overly large d can reduce model accuracy,.Source: C. Design Details · p. 6

IV. APPROXIMATE ATTENTION · A. Overview

A^3 treats attention as approximate content-based search: because many softmax weights are near zero and often do not affect accuracy, its approximation selects likely high-score candidates before query-time scoring. Preprocessing the key matrix shifts work out of query answering, with limited overhead when reused across multiple queries.

A. Overview

  • Attention computes query-to-key similarities, converts them to softmax weights, and returns a weighted sum of the corresponding value rows.Source: A. Overview · p. 6
  • Softmax amplifies a few large scores, leaving many candidates with near-zero weights that often do not contribute to model accuracy.Source: A. Overview · p. 6
  • Treating near-zero weights as zero avoids including those candidates in softmax and the following weighted-sum computation.Source: A. Overview · p. 6
  • A^3 approximates attention by selecting candidates likely to score highly without computing their scores, avoiding work on low-score key rows.Source: A. Overview · p. 6
  • The algorithm preprocesses the key matrix before query answering, when the query becomes available, to reduce operations on the query-time path.Source: A. Overview · p. 6
  • Preprocessing overhead is limited when multiple queries reuse a key matrix; BERT models may use the same matrix for 320 queries.Source: A. Overview · p. 6

B. Base Greedy Candidate Search · C. Efficient Greedy Candidate Search

A3 approximates dot-product scores by iteratively accumulating extreme component products and selecting positive-score rows as candidates. Its efficient version preprocesses key columns and uses priority queues to make query-time selection scale with M, with M controlling the performance–accuracy tradeoff.

B. Base Greedy Candidate Search

  • Large positive component products suggest large dot products, so the base search uses component-level extremes to estimate scores and selects rows with positive greedy scores as candidates.The selected candidates are passed to the base A3 dot-product module.Source: B. Base Greedy Candidate Search · p. 6
  • The base algorithm iteratively adds the kth-largest and kth-smallest matrix entries to row scores for M iterations, but sorting all nd entries costs O(nd log nd).Full dot-product computation takes O(nd), making the base search’s sorting-based complexity unattractive.Source: B. Base Greedy Candidate Search · pp. 6–7

C. Efficient Greedy Candidate Search

  • The efficient search preprocesses the key matrix by sorting each column, then performs candidate selection when a query arrives; preprocessing can be amortized across repeated queries.Although some models cannot preprocess off the critical path, BERT and Transformer self-attention reuse the same key matrix for multiple queries, such as 320 times.Source: C. Efficient Greedy Candidate Search · p. 7
  • For each query component, the efficient search initializes per-column max and min pointers according to the component’s sign, then inserts their query-weighted entries into priority queues.The queues track entries using their row and column IDs.Source: C. Efficient Greedy Candidate Search · p. 7
  • Each iteration pops the current extreme from maxQ and minQ, adds positive or negative values to the greedy score array, advances the corresponding column pointers, and reinserts entries.After M iterations, positive-score rows become candidates; the search skips minQ when the cumulative selected sum is negative to avoid selecting too few candidates when overall similarity is low.Source: C. Efficient Greedy Candidate Search · p. 7
  • Candidate selection takes M log d time, and increasing M as N grows can maintain reasonable accuracy while exposing a hyperparameter to balance performance and accuracy.The paper evaluates candidate-set estimation across different M in Section VI-B.Source: C. Efficient Greedy Candidate Search · pp. 7–8

D. Post-scoring Approximation · V. A3 WITH APPROXIMATION

A3 reduces post-scoring work by excluding low-scoring candidates from softmax and weighted-sum computation, then adds hardware modules for candidate selection and post-scoring approximation to its base accelerator.

D. Post-scoring Approximation

  • A candidate-scoring approach sorts rows by dot-product score and passes only top-scoring rows to the next steps.Those next steps are softmax and the final weighted sum.Source: D. Post-scoring Approximation · p. 8
  • The approximation targets low dot-product scores because most become near-zero after softmax and minimally affect the final outcome.After selecting candidate rows, A3 computes their full scores as key-row/query dot products, which feed softmax weights for the weighted sum.Source: D. Post-scoring Approximation · p. 8
  • Rows scoring more than threshold t below the top row are excluded from softmax and weighted-sum computation.Their post-softmax weights are at least e^t times smaller than the top row’s weight.Source: D. Post-scoring Approximation · p. 8
  • Dynamic post-scoring approximation adapts inclusion to score distributions, unlike a static number of top-scoring rows.Low-variance high scores may warrant including all such rows, while one high score among many low scores may not warrant including the low-scoring rows.Source: D. Post-scoring Approximation · p. 8
  • The paper uses T = 100 * (1/e^t) instead of t, with T specifying the minimum post-softmax weight as a percentage of the maximum for inclusion.A row is excluded when its score is more than t below the top-scoring row’s score.Source: D. Post-scoring Approximation · p. 8

V. A3 WITH APPROXIMATION

  • A3 adds hardware accelerator modules for candidate selection and post-scoring approximation to the base attention-mechanism accelerator.Connecting these modules to the base accelerator completes A3 with approximation capability.Source: §V · p. 8

A. Candidate Selection Module · B. Post-Scoring Selection Module

A3 specializes both candidate selection and post-scoring selection: it pipelines greedy selection over sorted keys, then filters dot-product candidates using a threshold before exponent computation.

A. Candidate Selection Module

  • The candidate-selection module buffers sorted key-matrix values and their original row indices in SRAM, alongside per-column pointers, product queues, comparators, and greedy-score registers.The module is designed to accelerate the algorithm in Figure 7 using customized hardware and parallelism.Source: A. Candidate Selection Module · p. 8, Figure 9 · p. 9
  • The candidate-selection pipeline precomputes component products to break loop dependencies and sustains one selection iteration per cycle.It initializes with c × d precomputed results in d circular queues and refills the selected column c cycles later.Source: A. Candidate Selection Module · p. 8
  • Initialization fills two sets of 4 × d product-buffer entries with 8d multiplications, borrowing d multipliers each from the base A3 dot-product and output-computation modules.After initialization, the candidate-selection module performs only two multiplications per cycle.Source: A. Candidate Selection Module · pp. 8–9
  • A d-way comparator tree selects the maximum and minimum component products in one cycle, reducing the selection algorithm’s complexity from O(Mlogd) to O(M).In steady state, the module takes M plus a few extra cycles for the iterative selection portion.Source: A. Candidate Selection Module · p. 9
  • After iterative selection, the module scans greedy-score registers in groups of up to 16 contiguous entries and sends positive-score row IDs onward.The module updates greedy scores during each candidate-selection iteration.Source: A. Candidate Selection Module · p. 9

B. Post-Scoring Selection Module

  • Integrated at the beginning of exponent computation, post-scoring selection identifies top entries by comparing remaining dot-product results with the maximum.It computes differences between the maximum dot-product value and the remaining entries.Source: B. Post-Scoring Selection Module · p. 9, Figure 10 · p. 9
  • The post-scoring module processes up to 16 entries per cycle and forwards entries whose difference from the top score exceeds threshold t, using 16 subtractors and comparators.Forwarded entries proceed to the exponent-computation module.Source: B. Post-Scoring Selection Module · p. 9

C. A3 HW with Approximation Support · VI. EVALUATION · A. Workloads

A3 uses staged candidate selection and scoring to approximate attention, with latency and throughput governed by its selection stages; evaluation covers three attention-based workloads with different sequence sizes.

C. A3 HW with Approximation Support

  • A3 selects candidate rows, computes their dot products, filters rows for exponentiation, and forms the output as a weighted sum.The post-scoring selector is integrated with the exponent computation module.Source: C. A3 HW with Approximation Support · p. 9
  • With C candidates selected over M iterations and K rows selected post-scoring, A3’s total latency is M + C + K + K + α cycles.α is a constant.Source: C. A3 HW with Approximation Support · p. 9
  • Throughput is limited by candidate selection at approximately M cycles, because iterations may update the same row and only positive-greedy-score rows are selected.Source: C. A3 HW with Approximation Support · p. 9

A. Workloads

  • The workloads include MemN2N on bAbI QA, which tests twenty question-answering task types using attention to find relevant statements.Source: A. Workloads · p. 9
  • Evaluation also uses KV-MemN2N on Wikimovies and BERT base, whose Transformer self-attention is tested on SQuAD v1.1.Wikimovies involves answering questions about movies after comprehending Wikipedia excerpts.Source: A. Workloads · pp. 9–10
  • All workloads use embedding dimension d = 64, while their respective n values are bAbI average 20 and maximum 50, Wikimovies average 186, and SQuAD maximum 320.For bAbI, n counts statements per query; for Wikimovies, potentially relevant knowledge; and for SQuAD, input passage and question length in words.Source: A. Workloads · p. 10

B. Accuracy Evaluation · C. Performance Results

A3’s approximation schemes trade accuracy for smaller selections and improved attention performance, while configurable candidate and post-scoring thresholds let users choose the tradeoff. The evaluation reports accuracy across three workloads and compares A3 throughput and latency with conventional hardware.

B. Accuracy Evaluation

  • Accuracy is evaluated for inference on test inputs using accuracy for bAbI QA, MAP for Wikimovies, and F1 score for SQuAD.The approximation model is integrated with the workloads’ official or endorsed open-source implementations.Source: B. Accuracy Evaluation · p. 10
  • Reducing candidate-selection iteration count M changes both the number of selected candidates and accuracy; selecting more candidates increases accuracy but reduces approximation benefits.The evaluation varies M from n to 1/8n.Source: B. Accuracy Evaluation · p. 10, Figure 11 · p. 10
  • A relatively high post-scoring threshold T, such as 10%, can retain decent accuracy while selecting fewer entries, because low-weight rows can be ignored.Higher T is a more aggressive approximation; lower T is more conservative.Source: B. Accuracy Evaluation · p. 10, Figure 12 · p. 10
  • The conservative scheme loses around 1% accuracy, while the aggressive scheme loses an extra 8% but selects substantially fewer entries; M and T are configurable to tune this tradeoff.Aggressive approximation also misses more true top 2 entries for bAbI or top 5 entries for Wikimovies and SQuAD, and may not be practical on its own.Source: B. Accuracy Evaluation · p. 10, Figure 13 · p. 10
  • Using f = 4 fractional bits causes less than 0.1% accuracy degradation across all workloads.The hardware pipeline is designed to avoid additional precision loss across its stages.Source: B. Accuracy Evaluation · p. 11

C. Performance Results

  • A3 achieves orders-of-magnitude higher throughput than the Intel Xeon CPU on MemN2N and KV-MemN2N; 67 conservative approximate A3 units can exceed GPU throughput on BERT.A3 has lower throughput than the GPU on BERT when using one unit, but multiple units can scale near-perfectly; the passage attributes this partly to GPUs underutilizing resources on the workload’s small matrices.Source: Figure 14 · p. 11, C. Performance Results · p. 11
  • Both approximate A3 configurations improve latency over base A3, while aggressive approximation offers more speedup than conservative approximation at a relatively low or moderate accuracy cost.For BERT, amortized preprocessing reduces throughput by 7% (conservative) or 24% (aggressive) and increases latency by <11%; A3 retains significant benefits with this overhead included.Source: C. Performance Results · p. 11, Figure 14 · p. 11

D. Area, Power, Energy and Test Chip

A3 combines compact area and low power with energy-efficiency gains from approximation, and the section describes a scaled-down test chip that implements its functionality.

Methodology

  • A3 was implemented in Chisel, functionally verified, and synthesized for 1GHz using the TSMC 40nm standard cell library.The workload sizing used n = 320 and d = 64, with number representation parameters i = 4 and f = 4.Source: D. Area, Power, Energy and Test Chip · pp. 11–12

Area

  • A3 uses less than 2.082mm2 area, while the CPU and GPU baselines are respectively 156× and 391× larger in die area.A3 was synthesized at 40nm, so the effective area difference is larger when accounting for the baselines’ process nodes.Source: D. Area, Power, Energy and Test Chip · p. 12

Energy and Power

  • A3 improves energy efficiency by multiple orders of magnitude over conventional hardware: over 10^4× on CPU and 10^3× on GPU.The comparison assumes CPU and GPU power consumption equal to their TDPs; approximation further saves energy by avoiding unnecessary computation.Source: D. Area, Power, Energy and Test Chip · p. 12, Figure 15 · p. 12
  • A3 consumes less than 100mW at full utilization and still less on real workloads because approximation creates pipeline imbalance.The passage compares this with the Intel Xeon CPU’s 115W TDP and NVIDIA Titan V GPU’s 250W TDP.Source: D. Area, Power, Energy and Test Chip · p. 12
  • Base A3 spends most energy on output computation, whereas approximate A3 spends most on candidate selection after reducing rows for other modules.Figure 15b presents the energy breakdown for base A3 and conservative and aggressive approximate A3.Source: D. Area, Power, Energy and Test Chip · p. 12, Figure 15 · p. 12

Test Chip

  • The scaled-down TSMC 40nm LP test chip fits a full-functionality A3 implementation in a 1mm2 die, with a 0.36mm2 core sized for MemN2N.It was functionally verified and communicates with a host ARMv8 CPU via a custom JTAG-like serial interface over 3.3V GPIO pins.Source: D. Area, Power, Energy and Test Chip · p. 12, Figure 16 · p. 12

VII. RELATED WORK · VIII. CONCLUSION

Related work places A3 among attention applications, approximate similarity-search methods, and neural-network accelerators. The conclusion argues that attention merits dedicated acceleration alongside better-optimized neural-network primitives.

VII. RELATED WORK

  • Attention is used in natural-language tasks including translation, and question answering,,,,,, as well as computer-vision tasks such as image classification,,, [53].The passage also lists language inference, summarization, document classification, visual question answering, image captioning, action recognition, and saliency detection.Source: §VII · p. 12
  • Neural Turing Machine and Differentiable Neural Computer use attention as long-term memory, and A3 applies to many such neural-network models.Source: §VII · p. 12
  • Prior similarity-search methods use hashing, trees, clustering,,, or greedy iterative search; attention-specific approximate searches, were implemented in software.A3 presents a hardware-friendly approximate-search algorithm and accelerator.Source: §VII · p. 13
  • CNN and RNN accelerators exploit parallelism and data-movement optimization, while prior sparsity accelerators target zeros in CNN/RNN weights or activations rather than approximation in dense attention matrices.Source: §VII · p. 13
  • Approximate neural-network hardware includes SnaPEA for convolution, RnR for reduce-and-rank operations, and approximate MAC units,,.Source: §VII · p. 13

VIII. CONCLUSION

  • A3 targets the relatively less-optimized attention primitive and uses software-hardware co-design to achieve orders-of-magnitude energy-efficiency improvement over conventional hardware.The conclusion invokes Amdahl’s Law: optimizing other neural-network primitives does not remove the need to accelerate attention.Source: §VIII · p. 13

Related papers

Loading 2002.10941v1…