PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

Jungseob Lee, Seungyoon Lee, Seongtae Hong, Sugyeong Eo, Heuiseok Lim

arXiv:2609.33889v1cs.LGcs.CLcs.PF

TL;DR

Single-sequence decoding reads fixed projection weights and a KV cache that grows with context, complicating comparisons between activation and KV-cache sparsity. The paper derives a byte crossover and speedup bounds, then evaluates the branches and their composition with matched attention kernels. Projection sparsity leads at short contexts and KV sparsity at long contexts; their composition improves on the best single branch under matched perplexity budgets.

  • Problem

    Activation sparsity reduces projection-weight reads and KV-cache sparsity reduces cache reads, but their reported speedups are difficult to compare across context lengths, dense attention kernels, and quality costs.

    Source: §1 · p. 1

  • Method

    The paper derives a byte crossover and ideal speedup bounds from model dimensions and keep ratios, then times both branches and their composition after real-text prefill with a shared split-K attention kernel.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    Short contexts favor the projection branch and long contexts the KV branch; composition is 25 to 26% faster than the best single branch on RTX A6000 and 14 to 24% faster on A100 at matched perplexity budgets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    With measured kernel costs included, the byte account predicts measured latency crossings within 4.1K tokens, while attention-scored selection matches dense decoding on passkey and multi-key placements up to 127K tokens.

    Source: §7 · p. 9

    The studied branches change what each step reads but leave unselected weights in memory and the full cache allocated and updated.

    Source: §3.2 · p. 4

Abstract

from arXiv · show

At each step, decoding one sequence with a large language model rereads the projection weights, whose traffic is fixed, and the key-value (KV) cache, whose traffic grows with context. Activation sparsity trims the first term and KV-cache sparsity the second, yet their reported speedups are hard to compare because each depends on context length and on the dense attention kernel it is measured against. We derive a byte crossover, the context length at which the two savings are equal, together with ideal speedup bounds for each branch and for their composition, from model dimensions and keep ratios alone. We then time both branches and their composition from 2K to 128K tokens on two GPUs after a dense prefill of real text, with dense and sparse modes reading the cache through the same split-K attention kernel. The projection branch leads at short context and the KV branch at long context, with speedups that follow their byte bounds up to fixed kernel costs. Adding these costs, measured in separate sweeps, lets the byte account predict the measured crossings of three keep-ratio pairs, a second model, and a second GPU to within 4.1K tokens. Timing the dense baseline with masked instead of split-K attention inflates the apparent speedup of the same KV policy about fivefold. An attention-scored KV selection answers the same passkey and multi-key placements as dense decoding up to 127K tokens, whereas a KV window misses most of them. Under matched perplexity budgets, activation sparsity composed with this selection decodes 14 to 26% faster than the best single branch on both GPUs. Code is available at https://github.com/js-lee-AI/ByteCross.

1 INTRODUCTION

Single-sequence decoding reads fixed projection weights and a KV cache that grows with context, making the two sparsity branches difficult to compare across context lengths and dense-kernel baselines. The paper derives byte-based predictions and evaluates both branches and their composition under a shared attention kernel.

1 INTRODUCTION

  • Projection traffic is fixed while KV-cache traffic grows linearly with context, so their savings vary differently with length and reported speedups are difficult to compare across kernels and quality costs.The dense attention kernel also changes the apparent gain of the same KV policy about fivefold.Source: §1 · pp. 1–2
  • A byte account derives the branches’ crossover and ideal speedup bounds from model dimensions and keep ratios, then times both branches and their composition from 2K to 128K tokens.Dense prefill uses real text, and dense and sparse modes read the cache with the same split-K attention kernel.Source: §1 · p. 2
  • The projection branch leads at short context, the KV branch at long context, and adding measured kernel costs predicts latency crossings within 4.1K tokens.The byte account places the KV selection within 1% of its byte bound on RTX A6000.Source: §1 · p. 2
  • At matched perplexity budgets, the composition is 25 to 26% faster than the best single branch on RTX A6000 and 14 to 24% faster on A100.Its attention-scored KV selection passes passkey and multi-key tests up to 127K tokens.Source: §1 · p. 2

2 RELATED WORK

Related work addresses decoding traffic through projection-side sparsity, KV-cache sparsity, and methods that improve execution paths. These approaches reduce different parts of decoding, but prior work had not measured projection-side and KV-cache sparsity against each other across context lengths.

Memory-bound decoding and serving

  • Serving approaches improve dense decoding through batching, cache storage, fused kernels, scheduling, and grouped-query attention, while weight-only quantization reduces weight traffic.These approaches retain the two-term traffic structure on which the crossover analysis operates.Source: §2 · pp. 2–3

Projection-side sparsity

  • Projection-side methods predict, threshold, or otherwise exploit active channels to reduce fixed projection-weight reads (Liu et al., 2023b; Mirzadeh et al., 2024; Song et al., 2024b; 2025; Wang et al., 2024).Variants include direct thresholding, learned predictors, offloading, and batched sparsity.Source: §2 · p. 3

KV-cache sparsity

  • KV-cache methods reduce growing cache traffic through eviction, windows, prompt compression, query-relevant selection, sparse attention, speculative verification, or quantization.Their effective keep ratio enters the byte account, while selection, packing, and cache updates add costs of their own.Source: §2 · p. 3

Execution-path methods

  • Prior work had not measured projection-side and KV-cache sparsity against each other along the context axis.Execution-path methods such as speculative decoding and adaptive layer selection can compose with either branch because each branch reduces bytes read per step.Source: §2 · p. 3

3 THE BYTE CROSSOVER

The byte crossover compares the projection and KV-cache reads removed by sparsity, using fixed projection traffic and context-growing cache traffic. From model dimensions and keep ratios, it yields a crossover and ideal speedup bounds, while the paper’s branch definitions and assumptions specify what those predictions cover.

3 THE BYTE CROSSOVER

  • The byte account equates the bytes removed by the two branches to obtain their crossover and derives ideal speedup bounds from the total bytes read and bytes removed.The crossover assumes both branches run at their byte bounds; the bound falls with context for projection sparsity and rises for KV sparsity.Source: §3 · p. 3, §3.3 · p. 4

3.1 DECODE-STEP MEMORY TRAFFIC

  • Only KV-cache reads grow with context; each decode step loads projection weights once, while other reads include the language-model head, an embedding row, and normalization weights.For a Llama-3.1-8B layer, projection weights account for 436 MB per step, versus 4096n cache bytes.Source: §3.1 · pp. 3–4

3.2 SPARSE BRANCHES

  • Projection sparsity keeps a fraction rP of activation channels and saves context-independent projection bytes, while KV sparsity reads a fraction rKV of cache entries and saves bytes that grow with context.Neither branch discards state: unselected weights remain in memory, the full cache remains allocated and continues receiving entries, and branches can switch between steps.Source: §3.2 · p. 4

3.3 CROSSOVER LENGTH AND SPEEDUP BOUNDS

  • The crossover n* is where the branches’ byte savings match; Table 1 reports thresholds at 50% projection keep and 30% KV keep for all projections and feed-forward projections alone.The threshold depends on model dimensions, keep ratios, KV layout, and precision; the 70B model’s crossover lies beyond its 128K window at the listed keep ratios.Source: §3.3 · p. 4, Table 1 · p. 4
  • A latency gate accounts for fixed kernel costs and quality admissibility, while a fixed KV token budget substitutes rKV = b/n into the crossover calculation.For a batch, shared projection reads depend on the union of activated channels, increasing the effective projection keep ratio and reducing each sequence’s crossover.Source: §3.3 · p. 5

4 WHERE THE BRANCHES CROSS

The section compares projection and KV sparsity across context lengths, using byte bounds and measured timings to locate when each branch is advantageous. It also shows that the dense attention baseline can substantially change the apparent gain of a KV policy.

4.1 SETUP

  • The projection branch uses TEAL activation sparsity at 50% keep, while the KV branch uses attention-scored selection at 30% keep after dense prefill.The KV method keeps four sink tokens and the highest-scoring entries per KV head, then gathers them into a contiguous buffer.Source: §4.1 · p. 5
  • All modes are timed in FP16 at batch size 1 with compiled decoding through FlashAttention’s split-K kernel, after dense WikiText-2 prefill, using repeated 50-step decodes.Timing used A100-SXM4 80 GB and RTX A6000 48 GB GPUs; the RTX A6000 sweep ended at 64K because its 128K cells did not fit in memory.Source: §4.1 · p. 5

4.2 SPEEDUP ACROSS CONTEXT

  • The projection branch leads at short context, while KV selection overtakes it between 32K and 64K on A100 and reaches 1.54× at 128K.At 2K, projection speedup is 1.36× on A100 and 1.67× on RTX A6000; KV selection runs within 6% of its byte bound on A100 and 1% on RTX A6000.Source: §4.2 · p. 5, Figure 3 · p. 6
  • From 8K tokens onward, composition is fastest on both GPUs, reaching 1.91× at 128K on A100 and 2.00× at 64K on RTX A6000.The faster single branch reaches 1.54× on A100 and 1.36× on RTX A6000 at those respective lengths.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3 LOCATING THE CROSSING

  • Cost-aware predictions place A100 crossings within 4.1K tokens and RTX A6000 crossings within 0.6K tokens of measurements.Using A100 kernel costs with the RTX A6000 dense-step time places its two crossings within 3.5K tokens; crossing order follows byte crossovers, while offsets depend on kernel costs and bandwidth.Source: §4.3 · pp. 6–7, Table 2 · p. 6

4.4 THE ATTENTION BASELINE

  • At 128K, the same KV window is 6.92× faster with masked attention and 2.82× with fused attention, versus 1.29× with split-K attention.The split-K result is below the 1.60× byte bound; masked and fused baselines use 5% and 12% of peak bandwidth, versus 78% for split-K.Source: §4.4 · p. 7, Figure 5 · p. 7
  • The projection branch transfers to Mistral-7B-v0.3, raising RTX A6000 throughput from 48.0 to 82.2 tok/s at 50% keep, a 1.71× speedup.In batching, the reported crossover falls to 3.0K tokens at batch size eight, well below n∗/4.Source: §4.4 · p. 7

5 QUALITY, COMPOSITION, AND DISPATCH

The section tests KV selection against retrieval requirements, then compares branches at matched perplexity budgets and evaluates composition and dispatch. Composition outperforms the best single branch under qualifying budgets, while a latency-calibrated gate improves the synthetic request mix over always using projection sparsity.

5.1 RETRIEVAL AS AN ADMISSIBILITY TEST

  • At 127K, attention-scored selection matches dense decoding’s multi-key placements, while a KV window answers only placements inside its retained region.The selection answers every placement at 32K and 64K, alone and composed with projection sparsity; the 30% window keeps more entries than the 20% selection yet misses most placements.Source: §5.1 · pp. 7–8, Table 3 · p. 7

5.2 THE QUALITY-MATCHED CROSSOVER

  • 35.7K, 48.1K, and 57.8K are the equal-PPL byte crossovers for budgets of 0.2, 0.5, and 1.0, below the nominal 74.3K.At 10% KV keep, selection changes perplexity by 0.06 and still passes both retrieval tests; at 20% keep, its perplexity change is under 0.01.Source: §5.2 · p. 8, Figure 6 · p. 8

5.3 COMPOSITION UNDER A QUALITY BUDGET

  • The quality-budget comparison selects and re-times the fastest configuration and fastest single branch within each budget across 16 settings and eight fixed 32K WikiText-2 windows.The selected settings are re-timed in five independent blocks.Source: §5.3 · p. 8, Table 4 · p. 8
  • 25–26% faster than the best single branch from a 0.2 perplexity budget upward, selected compositions also lead by 14–24% on A100.Each of five RTX A6000 blocks shows at least a 21% gain; every selected configuration passes both retrieval tests.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 DISPATCH ON REQUEST MIXES

  • On the synthetic request mix, the latency-calibrated gate reaches 1.40× on A100 and 1.72× on RTX A6000 over dense decoding.Always composing reaches 1.40× and 1.69×, so nearly all of the gate’s gain comes from composition.Source: §5.4 · p. 9

6 IMPLICATIONS FOR DEPLOYMENT

For deployment, the paper proposes combining quality-matched byte estimates with timings on the deployed attention kernel. It also explains how batching, quantization, and model dimensions shift the crossover.

6 IMPLICATIONS FOR DEPLOYMENT

  • The request-mix comparison reports decode speedups over dense decoding and charges selection-using requests for the scoring pass.The table caption identifies the comparison scope but does not provide specific policy results.Source: Table 5 · p. 9
  • A practical deployment sequence is to compute a quality-matched byte crossover, time both branches and their composition at several lengths, then admit KV settings only after retrieval tests.On A100, a 1 ms kernel-cost difference moves the latency crossing by roughly 17K tokens.Source: §6 · p. 9
  • With batch sharing, the per-sequence crossover falls to a few thousand tokens, favoring KV-cache sparsity from short contexts onward.Weight quantization shifts the crossover closer as well; larger feed-forward layers can push it beyond 128K tokens for Llama-3.1-70B at the studied keep ratios.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents the byte account as a way to predict branch crossovers and speedup bounds, with measured crossings predicted within 4.1K tokens when kernel costs are included. It recommends reporting byte bounds and dense-baseline attention paths, and testing retrieval before admitting KV configurations.

7 CONCLUSION

  • Within 4.1K tokens, the byte account predicts every measured latency crossing across the tested keep-ratio pairs, models, and GPUs when branch kernel costs are included.Measured speedups track byte bounds up to those fixed kernel costs after real-text prefill on a common attention kernel.Source: §7 · p. 9
  • An attention-scored KV selection passes passkey and multi-key tests up to 127K tokens that a KV window fails, and its composition with activation sparsity is fastest under budgets admitting projection sparsity.The selection matches dense decoding’s retrieval placements at 127K, where dense decoding itself misses four multi-key placements.Source: §7 · p. 9
  • The work measures publicly released models on public benchmarks and synthetic prompts, with no human subjects, personal data, or new datasets.The authors foresee no risks beyond those of the models themselves.Source: ETHICS STATEMENT · p. 9

A EXPERIMENTAL DETAILS

The appendix specifies the evaluated models, software, timing, and quality protocols, alongside implementation details for selection and threshold transfer. It also describes how quality-matched crossovers and synthetic request-mix costs are calculated.

A EXPERIMENTAL DETAILS

  • Llama-3.2-3B-Instruct has 28 layers and d=3072, with byte crossover at 35,109 tokens (34.3K) under 50% projection keep and 30% KV keep.Llama-3.1-8B-Instruct has a 128,256-token vocabulary.Source: A EXPERIMENTAL DETAILS · p. 15
  • The projection branch uses public TEAL thresholds in gpt-fast, while every attention mode reads cache through flash-attn 2.6.3 split-K attention.The scored selection gathers retained entries into a contiguous buffer; the window reads the cache in place and handles sinks in a separate call.Source: A EXPERIMENTAL DETAILS · p. 15
  • Timing cells use fresh processes, dense prefill of WikiText-2 in 512-token chunks, five warm-up steps, and five repeats of 50 decode steps.Cells are run in varied order within GPU blocks; campaigns use three blocks for sweeps and five for composition, while crossing campaigns use five A100 and four RTX A6000 blocks.Source: A EXPERIMENTAL DETAILS · p. 15
  • Decode-position perplexity scores the final 512 tokens of eight evenly spaced WikiText-2 windows after dense prefill of the preceding tokens.Task scores and chunked perplexity use plain completion rather than the chat template, so only paired dense-versus-sparse differences are meaningful.Source: A EXPERIMENTAL DETAILS · p. 15
  • Passkey trials place a six-digit code at five relative depths, while multi-key trials place four archive-linked codes and query one archive at those depths.Each test uses eight trials per depth; an answer giving another archive’s code is counted as a distractor.Source: A EXPERIMENTAL DETAILS · pp. 15–16
  • Scored selection ranks cache positions using attention from the last 64 prompt tokens, pooled across query heads and smoothed with a width-7 average.It forces sinks and the observation window into the retained set, then keeps top positions per KV head within the budget.Source: A EXPERIMENTAL DETAILS · p. 16
  • Transferred projection thresholds lower perplexity increases slightly in most windows and retain 51.6% of projection bytes at nominal 50% keep, versus 52.2% for transferred thresholds.The transferred thresholds are described as a slightly conservative stand-in for thresholds fitted to the evaluated model.Source: A EXPERIMENTAL DETAILS · p. 16
  • Quality-matched crossover equates byte savings after choosing each branch’s most aggressive tested keep ratio within the interpolated perplexity budget.Request-mix costs integrate measured latency along generated sequences and charge selection users a prompt-proportional scoring pass; the gate chooses the lowest tabulated cost among four configurations.Source: A EXPERIMENTAL DETAILS · p. 16

B PROJECTION-BRANCH SPEED AND TASK QUALITY

The section reports projection-branch speed and task quality at different keep ratios. Lower keep ratios decode faster as perplexity rises, while the gap between the FF-only and full variants widens with sparsity.

B PROJECTION-BRANCH SPEED AND TASK QUALITY

C DECODE-STEP TRAFFIC AND CROSSOVER VALUES

The section quantifies how projection and KV savings cross as context grows. The crossover depends on keep ratios and model dimensions, while the layer count cancels for models with the same layer dimensions.

C DECODE-STEP TRAFFIC AND CROSSOVER VALUES

  • At 50% projection keep and 30% KV keep, the crossover is 76,069 tokens for Llama-3.1-8B and Mistral-7Bv0.3, and 67,291 for Qwen3-8B.The layer count cancels, so these values hold for any depth with the same layer dimensions.Source: C DECODE-STEP TRAFFIC AND CROSSOVER… · p. 17
  • For Llama-3.1-8B, the crossover is 45,641 tokens at 70% projection keep and 106,496 tokens at 50% KV keep.Feed-forward projections alone give crossovers of 61,440 tokens for Llama-3.1-8B and 52,663 for Qwen3-8B at the stated keep ratios.Source: C DECODE-STEP TRAFFIC AND CROSSOVER… · p. 17
  • For Qwen3-8B, feed-forward weights account for 71.5% of reads at 512 tokens, while the cache accounts for 24.2% at 32K.At 32K, the KV saving reaches 62% of the feed-forward saving; the 51.4K feed-forward crossover lies beyond the model's 40K window.Source: C DECODE-STEP TRAFFIC AND CROSSOVER… · p. 17, Table 9 · p. 17

D HARNESS COSTS AND ATTENTION PATHS

Measured branch kernel costs shift latency crossings away from byte crossovers. Separate sweeps support cost-aware predictions, while the dense attention path can substantially change the apparent KV-policy gain.

D HARNESS COSTS AND ATTENTION PATHS

  • On A100-SXM4, the projection branch adds 1.9 to 2.3 ms beyond byte time, while selection adds at most 0.8 ms.On RTX A6000, the projection branch adds 1.0 to 1.5 ms and selection at most 0.3 ms; the window cost rises from 1.0 ms at 2K to 2.5 ms from 96K on A100-SXM4.Source: D HARNESS COSTS AND ATTENTION PATHS · p. 18, Table 10 · p. 18
  • The crossing model adds each branch's measured kernel cost to its byte-scaled dense time and solves for equal branch latency.Dense time is interpolated log-log and kernel costs linearly across sweep contexts.Source: D HARNESS COSTS AND ATTENTION PATHS · p. 18
  • Kernel costs place RTX A6000 latency crossings at 38.5K and 65.2K tokens, and the separate Llama-3.2-3B sweep follows the same cost pattern.The Llama-3.2-3B projection costs are 1.2 to 1.5 ms, while selection costs at most 0.44 ms.Source: D HARNESS COSTS AND ATTENTION PATHS · p. 18
  • The window runs at 21.6, 21.6, and 63.5 tok/s on the masked, fused, and split-K attention paths, respectively.The retention policy stays fixed while its implementation follows the attention path.Source: D HARNESS COSTS AND ATTENTION PATHS · p. 18, Table 11 · p. 18

E CROSSING CAMPAIGNS

  • Crossing campaigns time one keep-ratio pair across a context grid after real-text prefill, using five blocks on A100-SXM4 and four on RTX A6000.The reported crossing interpolates the mean difference between the last context where projection leads and the first where selection leads.Source: E CROSSING CAMPAIGNS · p. 19, Table 12 · p. 19, Table 13 · p. 20

F BATCHING

The section examines batching, long-context quality, composition, and dispatch. Under the reported quality budgets, compositions beat their single-branch comparators, and dispatch gains on the mixed workload come from composition.

F BATCHING

  • At B=1, the batched channel union is 0.52 rather than the nominal 0.5, placing its crossover slightly below the byte crossover.The union measures the fraction of projection weights at least one sequence in a batch activates.Source: F BATCHING · p. 20, Table 14 · p. 20
  • Dense throughput grows 22 to 25 times from one to 64 sequences on all three GPUs, while token latency grows less than three times.The batch sweep uses a Hugging Face generation loop with eager attention in FP16, so only scaling across batch size is comparable with the main results.Source: F BATCHING · p. 20, Table 15 · p. 20

G LONG-CONTEXT AND TWO-TURN QUALITY

  • At 127K, selection at 30% and 20% keep misses the same four multi-key placements as dense decoding, returning the same distractor code in each.Dense decoding answers all 40 second questions at both 32K and 64K, as do these selections and the stated composition.Source: G LONG-CONTEXT AND TWO-TURN QUALITY · p. 21

H COMPOSITION GRID AND INDEPENDENT TIMING

I DISPATCH ON REQUEST MIXES

  • Always composing and a calibrated gate each add speed over either single branch on the mixed workload, with dispatch gains attributed to composition.The workload combines chat, long-document questions, and agent loops.Source: I DISPATCH ON REQUEST MIXES · p. 22
Loading 2609.33889v1…