PaperlayerView exact revision on arXiv

Source-linked AI summary

SPARTA: A Sparse Ternary LLM Accelerator with Adaptive Attention and Unified Datapath for Long-Context Edge Inference

Yanzhen Zhou, Zhengyu Han, Qunyou Liu, Zheyu Yan, Weihong Xu, Cheng Zhuo

arXiv:2610.34824v1cs.ARcs.AI

TL;DR

Long-context edge inference is constrained by KV-cache traffic, and existing accelerators leave phase-specific sparsity and datapath unification insufficiently addressed. SPARTA combines phase-aware sparse dataflow, shared computation, and adaptive two-stage attention. It reduces KV traffic while reporting throughput and energy-efficiency gains over edge CPUs, GPUs, and other LLM accelerators, with task-dependent quality loss.

  • Problem

    Long-context inference on edge devices faces growing KV-cache traffic, while existing accelerators incompletely exploit phase-specific sparsity and often separate ternary linear and attention datapaths.

    Source: §I · p. 2

  • Method

    SPARTA combines phase-aware exploitation of heterogeneous sparsity, a shared fabric for ternary linear operations and bit-serial attention, and page-level Top-k followed by token-level Top-p pruning.

    Source: §I · p. 2

  • Results

    4.1–4.8× lower KV traffic, with task-dependent quality loss, alongside higher throughput and energy efficiency than edge CPUs, GPUs, and other LLM accelerators.

    Source: §I · p. 2

  • Takeaways & Limitations

    SPARTA demonstrates efficient long-context inference for ternary LLMs on edge devices through shared sparse computation and adaptive attention.

    Source: §VII · p. 12

    The quality loss from reducing KV traffic depends on the task.

    Source: §I · p. 2

Abstract

from arXiv · show

LLMs on edge devices are constrained by limited memory and bandwidth. Although 1.58-bit ternary weights greatly reduce model storage, long-context inference is increasingly bottlenecked by KV-cache traffic and attention. Existing accelerators also use separate datapaths for ternary linear layers and attention, incurring hardware and execution overhead. We present SPARTA, a sparse ternary LLM accelerator for efficient long-context edge inference. SPARTA integrates: (1) a phase-aware sparse dataflow exploiting heterogeneous sparsity across compute-bound prefill and memory-bound decode; (2) a unified compute fabric mapping ternary SpMM/SpMV and bit-serial attention onto shared processing elements to exploit zero-weight and zero-bit sparsity; and (3) an adaptive two-stage attention engine combining page-level Top-k selection to reduce K-cache accesses with token-level Top-p pruning to reduce V-cache traffic. SPARTA reduces KV traffic by 4.1-4.8x over full attention and 1.3-1.5x over a state-of-the-art sparse-attention scheme, with task-dependent quality loss. Compared with NVIDIA Jetson AGX Orin, SPARTA achieves 3.9x higher generation throughput and 8.3x higher end-to-end energy efficiency on average; versus AMD Ryzen 7 9700X, the gains are 5.0x and 81.7x. It also delivers 1.6-12.0x higher throughput, 2.0-4.2x higher energy efficiency, and 1.3-51.4x higher area efficiency than FIGLUT, TeLLMe, and MCBP.

I. INTRODUCTION

Long-context edge inference faces growing KV-cache traffic after ternary weights reduce weight traffic, while existing accelerators incompletely exploit phase-specific sparsity and often separate linear and attention datapaths. SPARTA combines phase-aware sparsity, a shared compute fabric, and adaptive attention; its reported results include lower KV traffic and higher accelerator efficiency.

I. INTRODUCTION

  • Ternary weights reduce storage and weight traffic, but the KV cache scales with context length and becomes the dominant per-token memory-traffic bottleneck.Recent ternary LLMs use 1.58-bit weights, retain 88–97% of full-precision intelligence, and use one-fifth the storage of 8-bit models.Source: §I · p. 1
  • Existing accelerators incompletely exploit sparsity across prefill and decode, use separate paths for linear and attention computation, and struggle to adapt sparse attention to variable distributions.Prefill is compute-bound, whereas decode is memory-bound; fixed Top-k budgets may discard important tokens or incur redundant KV accesses.Source: §I · pp. 1–2
  • SPARTA’s phase-aware sparse dataflow jointly exploits ternary-weight, attention-bit, and KV-access sparsity across inference phases.The design targets the complementary sparsity opportunities of compute-bound prefill and memory-bound decode.Source: §I · p. 2
  • SPARTA maps ternary linear operations and bit-serial attention onto a shared compute fabric, while adaptive page-level Top-k and token-level Top-p selection reduce K and V traffic.The shared fabric is intended for prefill, decode, linear layers, and sparse attention; the attention scheme selects pages and then prunes tokens.Source: §I · p. 2
  • SPARTA reduces KV traffic by 4.1–4.8× with task-dependent quality loss; it also reports higher throughput and energy efficiency than edge CPUs, GPUs, and other LLM accelerators.Against Jetson AGX Orin, generation throughput is 3.9× higher and end-to-end energy efficiency 8.3× higher; against Ryzen 7 9700X, the gains are 5.0× and 81.7×.Source: §I · p. 2

II. BACKGROUND AND MOTIVATION

Ternary weights reduce model storage and replace multiplication with add, subtract, or skip operations, but dense attention still accesses all KV data. Sparse-attention methods reduce those accesses through fixed-pattern, fixed-size Top-k, or probability-mass-based Top-p selection, each with distinct tradeoffs.

A. Ternary LLM Inference

  • Ternary linear layers use {−1, 0, +1} weights, cutting FP16 weight traffic by roughly 10× and replacing multiply-accumulates with add, subtract, or skip operations.Zero-valued weights enable zero skipping; attention remains higher precision to preserve model accuracy.Source: A. Ternary LLM Inference · p. 2
  • Dense attention scores every cached key and combines every value, so KV-cache quantization reduces bytes per entry but does not avoid accessing all KV data.Dense attention incurs quadratic complexity.Source: A. Ternary LLM Inference · p. 2

B. Sparse Attention

  • Fixed-pattern sparse attention retains query-independent sinks and local windows, but may discard distant, relevant tokens and lose accuracy under long contexts.These methods are hardware-friendly.Source: B. Sparse Attention · pp. 2–3
  • Content-aware Top-k selects a fixed-size set using exact or estimated relevance scores; a fixed page budget may over-fetch focused attention or discard context under diffused scores.Quest applies Top-k at the KV-page level.Source: B. Sparse Attention · p. 3
  • Top-p selects the smallest set covering a target probability mass, adapting retained-token counts to each query but requiring runtime selection and irregular data movement.Twilight combines coarse filtering with probability-mass-based pruning.Source: B. Sparse Attention · p. 3

C. Challenges and Opportunities

SPARTA targets phase-specific sparsity, datapath heterogeneity, and variable attention distributions across long-context inference. Its approach coordinates sparse execution across prefill and decode, unifies linear and attention operations, and adapts KV selection to attention distributions.

C1: Efficient Prefill and Decode Dataflows for Ternary LLMs

  • SPARTA coordinates ternary-weight, bit-level, and adaptive-attention sparsity across compute-bound prefill and memory-bound decode to improve end-to-end long-context inference efficiency.Attention sparsity can eliminate up to 90% of token accesses at long contexts.Source: C. Challenges and Opportunities · p. 3

C2: Unified Datapath and Efficient PE Design

  • SPARTA maps ternary GEMM/GEMV and bit-serial QK/PV onto one conditional-accumulation datapath, skipping zero weights and zero Q/P bits.Bank-local accumulation and conflict FIFOs handle irregular accesses without LUT or pattern reconstruction.Source: C. Challenges and Opportunities · p. 3
  • SPARTA's PE achieves up to 2.2× higher area efficiency and 1.5× higher energy efficiency than TENET and TeLLMe.Adapting MCBP's binary patterns to ternary weights yields area and estimated energy efficiencies of only 7.3% and 6.8% of SPARTA's.Source: C. Challenges and Opportunities · p. 3, Figure 4 · p. 3

C3: Adaptive Sparse Attention for Long-Context Inference

  • Attention tokens covering 90% of mass vary by three orders of magnitude, so fixed budgets can waste KV accesses or lose important tokens.Prior designs trade off learned prediction, full-context key scoring, fixed page budgets, and extra K-cache scans.Source: Figure 5 · p. 3, C. Challenges and Opportunities · p. 4
  • SPARTA combines page-level Top-k selection with histogram-based token-level Top-p pruning to adaptively reduce KV traffic.The proposed architecture targets low-complexity implementation on area-constrained hardware.Source: C. Challenges and Opportunities · p. 4

III. SPARTA DATAFLOW

SPARTA coordinates sparse execution across prefill and decode, then maps ternary linear layers and bit-serial attention onto a shared outer-product dataflow. This approach targets phase-specific compute and memory behavior while exploiting distinct forms of sparsity.

A. Phase-aware Sparse Dataflow

  • Prefill reuses bitmap-compressed ternary weights across prompt chunks, while decode streams weights and progressively selects pages, scores candidate tokens, and fetches retained values.Prefill attention scans cached chunks with online Softmax, retaining only its running maximum, normalizer, and output accumulator across chunks.Source: A. Phase-aware Sparse Dataflow · p. 4

B. Unified Sparse Execution for Linear and Attention

  • Ternary linear layers expose zero-weight sparsity, while bit-serial attention exposes zero bits in Q and P; SPARTA expresses both through conditional vector updates.The outer-product dataflow provides regular vector updates and reuses a selected operand across multiple lanes.Source: B. Unified Sparse Execution for Linear… · pp. 4–5
  • SpMM, SpMV, and bit-serial attention share the update z ← z + s 2^b v, with operator-specific operands, lane meanings, and choices of sign and shift.Ternary SpMM and SpMV use b = 0; attention uses shifted updates for set bits of Q or P.Source: B. Unified Sparse Execution for Linear… · p. 5

C. Adaptive Sparse Attention with Two-Stage Selection

SPARTA combines page-level Top-k selection with token-level Top-p pruning to reduce KV-cache traffic. The first stage limits K reads; the second narrows V reads among candidate tokens, with the target mass defined over that candidate set.

C. Adaptive Sparse Attention with Two-Stage Selection

  • Top-k selects pages from metadata before K gathering, then Top-p selects tokens from exact candidate QK scores; only retained tokens’ V data is gathered.Top-p’s target p is normalized over the Top-k candidate set and does not revisit excluded pages.Source: C. Adaptive Sparse Attention with… · pp. 5–6
  • The page budget is k = ⌈ρ_k⌈N/16⌉⌉, and the selected pages form a candidate set of ρ_kN tokens before Top-p pruning.For a 32K-token context at ρ_k = 1/4, the example budget is 512 pages.Source: C. Adaptive Sparse Attention with… · p. 5
  • Adding Top-p leaves K traffic unchanged while reducing V traffic to ρ_pρ_kN, and SPARTA’s two-stage scheme reduces threshold-search work to O(ρ_kN + B).SPARTA uses a fixed-size B-bin histogram, rather than repeatedly rescanning candidates to find the mass threshold.Source: C. Adaptive Sparse Attention with… · p. 5
  • For grouped-query attention, one page Top-k selection is shared per KV group, while Top-p remains per head and V is gathered for the union of retained sets.Sharing the candidate pages and K reads avoids multiplying a group’s K traffic by up to G.Source: C. Adaptive Sparse Attention with… · p. 6

A. Architecture Overview

SPARTA’s architecture combines a shared fabric for sparse linear and attention kernels with an adaptive attention engine. Its shared request interface keeps the PE array and output path unchanged when switching operations.

A. Architecture Overview

  • The shared fabric supports SpMM, SpMV, and attention, while a separate adaptive engine implements page Top-k and histogram Top-p selection.The architecture has eight compute slices connected to HBM2, with each slice containing the fabric, attention engine, operand buffers, and an activation-processing unit.Source: A. Architecture Overview · p. 6

B. Shared Fabric for Unified Sparse Execution

  • The shared fabric represents each vector update as a request carrying an opcode, shift, target bank and accumulator address, and operand vector.Switching operations changes request interpretation while the PE array and output path remain unchanged.Source: B. Shared Fabric for Unified Sparse… · p. 6
  • Skipping zero weights reduces ideal SpMM cycle complexity from O(MKN/(PL)) to O((1 − s_w)MKN/(PL)), while SpMV retains O(MK/(PL)).The passage estimates an ideal SpMM speedup of about 1.67× for a zero-weight fraction around 40%.Source: B. Shared Fabric for Unified Sparse… · p. 6
  • Skipping zero bits yields ideal QK and PV speedups of 3.2× and 5.7×, respectively, for zero-bit fractions of about 69% and 83%.SpMV and bit-serial attention use bank-aligned request streams that bypass the conflict FIFOs used for irregular SpMM requests.Source: B. Shared Fabric for Unified Sparse… · p. 7

C. Adaptive Sparse Attention Engine

The adaptive attention engine uses a fixed-size Bitonic sorter for page Top-k and a streaming histogram for token Top-p. The paper evaluates these mechanisms alongside its model, workload, and hardware baselines.

C. Adaptive Sparse Attention Engine

  • SPARTA reuses a fixed-size Bitonic sorter across page-score windows and merge levels, so longer contexts increase iteration count rather than sorter size.The selector buffers page scores in a 2048-entry SRAM and grows sorted runs up to its 512-entry capacity.Source: C. Adaptive Sparse Attention Engine · p. 7
  • The Top-p engine quantizes candidate score gaps into 64 bins and uses two fixed histogram passes to find the threshold.Only the scaling and binning latency grows with candidate count; the histogram, LUT, and reducer are fixed-size.Source: C. Adaptive Sparse Attention Engine · pp. 7–8
  • After thresholding, the engine compacts retained tokens into token-ID/P-bit pairs and prefetches only their V tiles for PV.A token’s V is skipped only when no head in its KV group retains it.Source: C. Adaptive Sparse Attention Engine · p. 8

A. Experimental Setup

  • Evaluation covers six ternary LLMs, perplexity and seven task benchmarks, with hardware comparisons spanning CPU, GPU, FPGA, and ASIC baselines.The setup includes Jetson AGX Orin and Ryzen 7 9700X systems, as well as FIGLUT, TENET, TeLLMe, MCBP, FACT, Energon, and LUT-DLA; the hardware simulation uses a trace-driven cycle-level simulator calibrated to RTL counts.Source: A. Experimental Setup · p. 8

B. Algorithm Evaluation

SPARTA evaluates adaptive sparse attention by balancing quality against computation and KV-traffic savings. At the default settings, page-level Top-k and token-level Top-p provide complementary traffic reductions, with quality effects varying by task.

Sparse Attention Parameters

  • At ρk = 1/4 and p = 0.90, four models remain within 1.3% of dense baselines, while the worst degradation among the others is 4.0%.These values motivate the selected default parameters.Source: B. Algorithm Evaluation · p. 9, Table III · p. 9

Quality across Models and Tasks

  • Reducing the page budget generally increases perplexity, while increasing p tends to recover quality within each budget.At the default settings, quality effects vary by task: long-document summarization and retrieval are more sensitive, and Top-k is disabled for short contexts.Source: B. Algorithm Evaluation · p. 9, Figure 12 · p. 9, Figure 11 · p. 9

Computation and Data Traffic Reduction

  • At ρk = 1/4 and p = 0.90, Top-k and Top-p together reduce KV traffic by 4.1× and operations by 5.9×.Top-k reduces K traffic, while Top-p further reduces V traffic and PV computation; Quest achieves 3.2× traffic reduction, and MCBP provides smaller savings after predictor overhead [14].Source: B. Algorithm Evaluation · p. 9, Figure 13 · p. 9

C. Hardware Performance

SPARTA’s sparse attention speeds decode and reduces KV reads, while attention takes a larger share of latency as context grows. Design-space sweeps select fabric and attention-engine settings that balance throughput, latency, and hardware cost.

C. Hardware Performance

  • 3.9× higher geometric-mean generation throughput and 8.3× higher end-to-end energy efficiency than the GPU; gains over the CPU are 5.0× and 81.7×, respectively.Both baselines use dense attention, so the gains combine architectural and sparse-attention benefits.Source: C. Hardware Performance · p. 9, Figure 14 · p. 10
  • As context grows from 2K to 32K, the average linear-layer share of prefill latency falls from about 82% to 28%.Projection and FFN overhead grows linearly with prompt length, whereas dense attention grows quadratically.Source: C. Hardware Performance · p. 10, Figure 15 · p. 10
  • At 32K, attention reaches 31–63% of decode TPOT, and two-stage sparse attention is 4.0–6.2× faster than dense attention.Top-k selection covers K generation except for Bonsai at 8K; Top-p covers V generation in all cases.Source: C. Hardware Performance · p. 10, Figure 16 · p. 10
  • Top-k reduces decode HBM reads by 3.2×; adding Top-p raises the reduction to 4.3–4.4× for BitCPM-3B and 4.9× for Bonsai-4B.The transferred bytes are 1.4–1.7% above the analytical bytes of selected metadata and tokens.Source: C. Hardware Performance · p. 10, Figure 17 · p. 10

D. Design Space Exploration

  • The design uses 16 PEs with 32 lanes; reducing lanes to 16 halves prefill throughput, while 32 PEs add 78% logic area without reducing decode latency.A conflict-FIFO depth of 8 reaches 96.7% of depth-32 throughput with one quarter of the entries.Source: D. Design Space Exploration · p. 11, Figure 19 · p. 11
  • A 64-entry Top-k window reduces TPOT by 8.8%, 13.9%, and 18.5% at 8K, 16K, and 32K; 64 histogram bins reduce TPOT by 6.1% at p = 0.90 and 7.7% at p = 0.95.Larger settings yield smaller additional reductions while increasing selector area or histogram SRAM cost.Source: D. Design Space Exploration · p. 11, Figure 20 · p. 11

E. Ablation Studies

SPARTA’s ablations attribute latency reductions to zero-weight skipping and the two stages of sparse attention. Comparisons with other accelerators show throughput and energy-efficiency gains across prefill and decode, with hardware-efficiency results also favoring SPARTA.

E. Ablation Studies

  • The final sparse-attention configuration differs from full attention by at most 0.61 PPL, while the histogram engine changes perplexity by at most 0.10 versus exact Top-p.The retained candidate fraction is 2.9–10.9% and decreases from 8K to 32K.Source: E. Ablation Studies · p. 11, Table IV · p. 11
  • Zero-weight skipping, Top-k, and Top-p reduce geometric-mean latency by 22%, 9%, and 5% relative to the preceding configuration, yielding a cumulative 1.48× speedup.Zero-weight skipping shortens prefill TTFT by 29%; Top-k and Top-p reduce decode TPOT by 26% and 16%.Source: E. Ablation Studies · p. 11, Figure 21 · p. 11

F. Comparison with SOTA LLM Accelerators

VI. RELATED WORK

SPARTA shares integer select-add updates across ternary linear computation and Q/P-expanded attention, then time-multiplexes prefill, decode, and sparse attention on one fabric. The conclusion reports throughput and energy-efficiency improvements over CPUs, GPUs, and state-of-the-art accelerators for long-context ternary LLM inference.

VI. RELATED WORK

  • SPARTA uses integer select-add updates across ternary linear and Q/P-expanded attention, and time-multiplexes prefill SpMM, decode SpMV, and sparse QK/PV on one fabric.Its adaptive sparse attention uses metadata-based page Top-k followed by exact candidate QK and token Top-p.Source: §VI · p. 12

VII. CONCLUSION

  • The authors report that SPARTA improves throughput and energy efficiency over CPUs, GPUs, and state-of-the-art LLM accelerators for long-context ternary LLM inference.Its shared fabric jointly exploits zero-weight, bit-level, and query-dependent KV sparsity across prefill and decode.Source: §VII · p. 12

VIII. AI USE

  • Claude and Codex assisted with drafting, editing, analysis, and plotting scripts; the authors reviewed, revised, and verified the AI-assisted content.The disclosure identifies the authors as responsible for the manuscript’s content.Source: §VIII · p. 16

Related papers

Loading 2610.34824v1…