PaperlayerView exact revision on arXiv

Source-linked AI summary

The KV Cache Is the New Memory Wall

Tejinder Singh

arXiv:2609.30854v1cs.DCcs.LGcs.PF

TL;DR

Long-context decoding shifts from weight traffic to KV-cache traffic, while inconsistent workloads, hardware, and quality metrics make mitigation results hard to compare. The paper develops a topology-aware analytical framework and fixed-protocol evaluation; reported findings distinguish precision and eviction trade-offs, with hardware and workload conditions shaping their relevance.

  • Problem

    Long-context decode becomes bounded by memory traffic, while inconsistent workloads, hardware, and quality metrics make reported KV-mitigation gains difficult to compare.

    Source: §1 · p. 4, §1.3 · p. 4

  • Method

    The paper analyzes decode traffic and hardware topology, classifies five mitigation domains, and evaluates representative methods under a fixed protocol.

    Source: §1.3 · p. 4, §6.1 · p. 19

  • Results

    Eight-bit storage is free and four-bit storage near-free with suitable layouts; at r = 0.25, eviction reports 96–99% retention on aggregate benchmarks, while retrieval probes degrade discontinuously.

    Source: §4.3 · p. 15, §4.4 · p. 16

  • Takeaways & Limitations

    Hardware topology and context-dependent traffic crossovers shape speedups, while precision and workload position sensitivity qualify the quality trade-offs of quantization and eviction.

    Source: §4.3 · p. 15, §4.4 · p. 16, §6.1 · p. 19

    Published evidence often lacks confidence intervals, consistent quality metrics, complete cost reporting, and disclosed workloads, limiting regime-based comparison.

    Source: §5.4 · p. 18

Abstract

from arXiv · show

Autoregressive LLM inference at long context is bounded by memory bandwidth, not arithmetic throughput, and the binding resource shifts from model weights to the Key-Value (KV) cache as sequence length grows. For Llama-3-70B in BF16, the 140 GB weight footprint exceeds the 80 GB HBM of a single accelerator, and one 128k-token sequence adds 42 GB of KV cache. Techniques that compress, evict, page, share, or offload KV state have proliferated, but reported gains use inconsistent workloads, hardware, and quality metrics, preventing cross-paper comparison. This SoK paper unifies the field analytically, with a protocol that strictly separates derived and reported claims. We derive closed-form arithmetic intensity as a decaying function of context length, parameterized by hardware topology for NVIDIA H100, NVIDIA B200, and AMD MI300X, including per-die bandwidth partitioning and the crossover lengths where KV traffic overtakes weight traffic. We classify the literature into five domains, quantization, token eviction, KV paging, prefix caching, and heterogeneous tiering, evaluating one method per domain at 128k context under a single protocol. The central finding is a three-regime structure: below a hardware-specific crossover, weight traffic dominates and KV compression yields negligible speedup; beyond it, KV traffic dominates and each domain trades quality for bandwidth savings approaching the roofline bound. Paging and prefix sharing are lossless but address capacity, not bandwidth. Quantization and eviction cut bandwidth directly, with degradation that accelerates below 4-bit precision and turns discontinuous for eviction on position-sensitive tasks. Tiering converts the bandwidth wall into an interconnect problem bounded by PCIe or NVLink rather than HBM. We close with design rules for selecting a compression domain given hardware, context length, and quality budget.

1 Introduction and the Decode-Phase Memory Wall

Long-context decode becomes a memory-traffic problem as the KV cache grows, while inconsistent evaluation conditions make mitigation results difficult to compare. The paper establishes an analytical and empirical framework for comparing five mitigation domains.

1.1 Decode-Phase Arithmetic Intensity

  • At batch size one, BF16 decode has 1 FLOP/B intensity against an H100 ridge of about 295 FLOP/B; batching shares weight reads but not KV reads.Attention intensity remains invariant to batch size, and KV traffic overtakes weight traffic as context grows.Source: §1.1 · p. 4

1.2 KV-Cache Growth with Context Length

  • A Llama-3-70B sequence adds 42 GB of KV state at 128k tokens, while its cache reaches 328 GB at 10^6 tokens.The BF16 model’s weights already exceed one 80 GB accelerator’s HBM; Figure 1 places the 70B and 405B H100 capacity crossings at 244k and 155k tokens.Source: §1.2 · p. 4, Figure 1 · p. 5

1.3 Five Fragmented Mitigation Families

  • Quantization, eviction, paging, prefix caching, and tiering use distinct mechanisms, but inconsistent workloads, hardware, and metrics leave reported gains difficult to compare or compose.The paper positions its analytical frame as a way to make the catalog comparable.Source: §1.3 · pp. 4–5

1.4 Comparability Deficiencies in the Published Evidence Base

  • Published comparisons lack a shared traffic model, account poorly for hardware topology, and report quality on incompatible scales, leaving most results not jointly interpretable.The paper notes that individual numbers may be correct while remaining difficult to interpret together.Source: §1.4 · p. 5

1.5 Systematization Goal and Contributions

  • The paper classifies five domains, derives context-dependent decode intensity and topology-aware crossover lengths, and evaluates one representative per domain on standardized 128k workloads.It also analyzes cross-cutting challenges and concludes with design rules keyed to hardware, context length, and quality budget.Source: §1.5 · pp. 5–6

2 KV Cache Anatomy and a Five-Domain Taxonomy of Compression

The section models KV-cache growth and organizes mitigation methods by the footprint factor they change or the memory tier in which bytes reside. It distinguishes direct byte reduction from capacity management and data relocation.

2.1 Exact KV-Cache Structure of Autoregressive Transformers

  • Decode appends each new key and value but reads the existing cache; architectural choices change per-token footprint, not its linear growth with sequence length.Across the listed architectures, GQA reduces per-token cost, while MLA’s DeepSeek-V2 design stores roughly five times less per token than GQA designs.Source: §2.1 · p. 6

2.2 A Unified Footprint Model

  • The unified footprint model assigns quantization to stored width, eviction to retention, paging to allocation waste, prefix caching to sharing, and tiering to residency.These domains scale orthogonal byte-accounting factors, though quality and kernel interactions do not factor as cleanly.Source: §2.2 · pp. 6–7, Figure 2 · p. 7

2.3 Domain I. Precision Reduction of Stored Keys and Values

  • Quantization reduces stored element width, but key outliers motivate asymmetric layouts and structure-aware methods; the bytes removed are not read from HBM on later decode steps.The cited methods use per-channel keys, per-token values, outlier isolation, or kernel co-design to address format-specific costs.Source: §2.3 · p. 7

2.4 Domain II. Token Eviction and Retention Policies

  • Eviction retains selected positions, but irreversible removal can cause discontinuous degradation when a task depends on arbitrary past positions.Read-path selection instead chooses a query-dependent subset at decode time, saving bandwidth without relieving cache capacity.Source: §2.4 · p. 7

2.5 Domain III. Paged Block Management of KV Memory

  • Paging allocates KV memory in fixed-size blocks, reducing fragmentation and enabling sharing while leaving the stored byte count unchanged.The cited production-trace comparison reports waste below 4% with paging versus 60–80% stranded under reservation-based allocation.Source: §2.5 · p. 8

2.6 Domain IV. Prefix Sharing Across Requests

  • Prefix caching reuses a prompt’s KV state across requests that share it, saving repeated encoding and making gains dependent on shared structure in the request stream.With no shared structure, the sharing factor remains σ = 1.Source: §2.6 · p. 8

2.7 Domain V. Heterogeneous Tiering of KV State

  • Tiering relocates KV bytes across memory levels rather than reducing them, so performance depends on interconnect bandwidth and whether reads are served from HBM.PCIe Gen5 x16 is about 50× below H100 HBM bandwidth, while NVLink at 900 GB/s is 3.7× below it.Source: §2.7 · p. 8

2.8 Orthogonality and Composition of the Five Domains

  • The domains compose multiplicatively in bytes, but quality costs do not; paging and prefix sharing are lossless, while the quality-budgeted choices are width, retention, and residency.The cited example combines 2-bit quantization, 50% retention, and sharing factor σ = 4 for a 1/64 footprint before tiering.Source: §2.8 · p. 8

3 Analytical Derivations of Context-Dependent Arithmetic Intensity

The section derives decode cost and arithmetic intensity from FLOP and byte accounting, then incorporates hardware topology and traffic crossovers. Its analysis identifies weight-bound, KV-bound, and capacity-bound regimes, with compression speedups depending on context length and placement.

3.1 Exact FLOP and Byte Accounting for a Decode Step

  • The decode-step cost model counts weight bytes once per step and KV bytes per sequence, with attention assuming one cache read.The section derives its analytical results from these FLOP and byte costs.Source: §3.1 · p. 9

3.2 Arithmetic Intensity as a Function of Context Length and Batch

  • At production batch sizes, decode intensity falls toward the KV floor as context grows; even 2-bit cache storage reaches only 64 FLOP/B, 4.6× below the H100 ridge.The weight-side intensity reaches the H100 ridge near batch 295, when Llama-3-70B is already KV-bound beyond 1.4k tokens.Source: §3.2 · p. 9, Figure 3 · p. 10

3.3 Hardware Topology and Per-Die Bandwidth Partitioning

  • Pinned KV placement halves B200 aggregate bandwidth and costs 8× on MI300X; topology-aware striping can recover aggregate bandwidth when the interconnect is sufficient.The analysis identifies KV page placement as a first-order performance decision on multi-die hardware.Source: §3.3 · pp. 10–11, Figure 4 · p. 11

3.4 Crossover Context Lengths

  • At production batch sizes 32–256, Llama-3-70B crosses into KV-bound operation at 13.4k–1.7k tokens; at 128k, KV traffic exceeds weight traffic.The section also finds that Llama-3-70B does not fit on one H100 with weights resident, while its capacity limit on B200 or MI300X is 159k tokens.Source: Table 4 · p. 11, §3.4 · p. 11, Figure 5 · p. 12

3.5 Worked Bounds and the Speedup Structure of Compression

  • Below s∗, weight traffic dominates and KV compression speedups approach one; above it, KV-byte reductions translate one-to-one into speedup until quality constraints bind.Beyond the capacity limit, tiering or multi-device parallelism becomes necessary.Source: §3.5 · p. 12
  • At 128k on B200, 2-bit quantization yields 1.25× speedup; at 512k it yields 1.91×, while adding 50% eviction reaches 2.05×.These are derived bounds, not measurements; the speedup changes with the workload’s position relative to the crossover.Source: §3.5 · p. 12, Table 5 · p. 13

3.6 Formal Statements of the Analytical Results

  • KV traffic’s share is s/(s + s∗(b)), so it reaches half at the crossover and the crossover falls as 1/b.In the memory-bound regime, speedup from KV-byte reduction rises with context and batch, approaching the byte-reduction factor at long context.Source: §3.6 · pp. 13–14
  • The placement model treats remote reads as fabric traffic without remote-HBM contention, making striped-placement estimates upper bounds that favor remote reads.The stated guidance is to stripe pages and avoid serving per-step reads from a remote tier.Source: §3.6 · p. 14

4 Comparative SoK Analysis Across the Five Compression Domains

The comparison fixes a shared model, hardware, context, and evidence protocol, then shows that each KV technique changes a different resource or trade-off. Lossy methods save decode bandwidth with quality costs that vary by precision and workload, while paging, prefix sharing, and tiering address capacity or reuse rather than offering equivalent decode-speed gains.

4.1 Workload and Metric Definitions

  • The protocol fixes Llama-3-70B, B200 with striped pages, and 128k context, separating derived traffic and speedup values from within-paper reported quality results.Speedups assume perfect kernel efficiency and full bandwidth realization; W2 and W3 ratios are evaluated as capacity-independent traffic regimes.Source: §4 · p. 15, §4.1 · p. 15, Table 7 · p. 16

4.2 The SoK Comparison Matrix

  • The same 2-bit quantization yields 1.25× at W1 and 4.82× at W2, while paging and prefix sharing change feasibility or stored bytes rather than per-step decode speed.Tiering converts a 25-second recompute into a sub-second fetch on reuse, an economic gain not captured by decode-bound metrics.Source: §4.2 · p. 15, Table 7 · p. 16

4.3 Precision Reduction Results

  • Each halving of KV width doubles the long-context intensity floor, but 2-bit storage holds only with structure-aware layouts and a full-precision residual window.Four-bit storage is near-free with per-channel keys and outlier isolation; naive round-to-nearest at 2-bit loses more than one perplexity point.Source: §4.3 · p. 15, Figure 6 · p. 17

4.4 Token Eviction Results

  • At 25 percent retention, heavy-hitter and observation-window policies retain 96 to 99 percent on aggregate benchmarks, while retrieval-style probes degrade discontinuously.Eviction therefore needs evaluation against workload position sensitivity; it composes with quantization through a combined byte factor.Source: §4.4 · p. 16

4.5 Paging, Prefix Sharing, and Tiering Results

  • Block-level paging keeps allocation waste under 4 percent, versus 60 to 80 percent for reservation-based allocation, increasing feasible batch size rather than reducing per-step bytes.Paging shows 1.00× in W1 and W2 decode-speed columns, while production throughput roughly doubles to quadruples.Source: §4.5 · p. 16
  • Prefix sharing lowers TTFT from 25.0 s cold to a derived 3.7 s at reported 90 percent hit rates, with zero quality cost.On W3, a shared 96k prefix is encoded once for eight requests; stored bytes fall 2.91×.Source: §4.5 · p. 16
  • On W4, fetching a 128k-token cache takes 0.66 s over PCIe Gen5, 0.16 s with a 4× codec, or 12 ms over NVLink with the codec, versus 25.0 s to recompute.Tiering loses when decode reads remote memory because PCIe bandwidth is 125× below B200 HBM; it is for reuse across steps, not reads within a step.Source: §4.5 · p. 16

4.6 Cross-Domain Synthesis and the Speedup-Quality Frontier

  • At W2, the speedup-quality frontier spans quantization and eviction, while lossless paging, prefix sharing, and tiering sit outside that comparison.The composed 2-bit, 50-percent-retention, and σ = 2.91 configuration approaches 46× in capacity terms and 6.6× in decode traffic terms; realizing composed gains requires kernels and schedulers that handle the layouts without materialization overhead.Source: §4.6 · pp. 16–17

5 Cross-Cutting Challenges and Open Research Directions

The paper maps five KV-cache domains across three traffic, capacity, and interconnect regimes, then identifies kernel, scheduler, and evidence gaps that constrain deployment. It proposes research directions for composing the domains and evaluating them consistently.

5 Cross-Cutting Challenges and Open Research Directions

  • No domain dominates across six capability dimensions, supporting treatment of the five domains as infrastructure layers rather than competitors.The section identifies recurring barriers in kernel support, scheduling and placement, and evaluation.Source: §5 · p. 17

5.1 The Three-Regime Map Revisited

  • Below s∗(b), weights dominate; between s∗(b) and scap, KV traffic dominates; beyond scap, tiering or multi-device parallelism is required.The paper attributes conflicting speedup reports to regime confusion.Source: §5.1 · p. 17

5.2 Compression-Aware Attention Kernels

  • Realized lossy-domain speedups fall below roofline bounds when kernels lack fused dequantization or efficient support for ragged eviction layouts.QServe measured 20 to 90 percent runtime overhead when dequantization falls back to generic CUDA cores.Source: §5.2 · p. 17

5.3 Scheduler Co-Design, Disaggregation, and Placement

  • Disaggregation moves a 42 GB cache per 128k request across the cluster fabric, while package placement can change bandwidth substantially.The text reports that striping versus pinning is worth 2× on B200 and 8× on MI300X, yet production block managers expose no die or stack affinity.Source: §5.3 · p. 18

5.4 Evaluation Gaps in the Published Evidence Base

  • Published evaluations often omit variance, use fragmented quality metrics, leave costs incomplete, and rely on synthetic or undisclosed workloads.These gaps impede identifying the workload regime that determines speedup claims.Source: §5.4 · p. 18

5.5 Open Research Directions

  • A unified paged layout should combine per-block precision and retention masks, with a success test of composed 2-bit, r = 0.5, σ = 4 at 128k.The target is performance within 20 percent of the 64× capacity bound.Source: §5.5 · p. 18
  • Other proposed directions are topology-aware placement, retention guarantees, link-matched codecs, and a standing public benchmark with separated evidence classes.The conclusion calls for treating KV state as a first-class structure with formats, placement, and transport co-designed across kernel, runtime, and cluster.Source: §5.5 · pp. 18–19

6 Conclusion and Design Rules

The conclusion frames decode as a three-regime memory problem and reports that the same compression method can have different speedups across batch sizes. It gives an ordered procedure for choosing domains and emphasizes that the equations are accounting identities, while specific values will change with models and accelerators.

6.1 Summary of Findings

  • The same 2-bit quantization yields 1.25× at batch one and 4.82× at batch thirty-two; composed 2-bit plus eviction reaches 6.6× in decode traffic and 46× in capacity.The conclusion divides the wall into traffic, capacity, and interconnect, with each domain addressing one cleanly.Source: §6.1 · p. 19

6.2 Design Rules for Domain Selection

  • Compute s∗(b) and scap first; below s∗ avoid cache compression, while above it quantize first and validate 2-bit layouts on position-sensitive probes.The rules also recommend 4-bit KV as near-free with per-channel keys and outlier isolation.Source: §6.2 · p. 20
  • Enable paging and prefix caching for their quality-neutral feasibility benefits, and evict only when the workload tolerates it.The rules default to r ≥ 0.5, require needle-style evaluation before going lower, and advise against eviction for retrieval-critical traffic.Source: §6.2 · p. 20
  • Beyond scap, tier or shard; use transport codecs for cross-link transfers and never serve per-step decode reads from a remote tier.The rules also recommend striping pages on multi-die packages and checking per-partition bandwidth.Source: §6.2 · p. 20
  • Re-evaluate the map when models, hardware, or workloads change, since those changes move the regime boundaries.For shared prefixes on W3, a warm request has derived TTFT of 8.8 s versus 25.0 s cold, or 2.8× lower TTFT, with stored bytes falling 2.91× at zero quality cost.Source: §6.2 · p. 20, Figure 9 · p. 20

6.3 Closing Remarks

  • The equations remain accounting identities over FLOPs and bytes, while their specific numerical inputs age as models and accelerators evolve.The closing argument calls for kernels, schedulers, and benchmarks to treat KV cache as a first-class data structure.Source: §6.3 · pp. 20–21

A Full Derivations

The appendix expands the main-text accounting to include lower-order terms and derives the prefill and topology models. It states that dropping those terms recovers the main-text equations without changing main-text quantities.

A Full Derivations

  • The appendix expands Section 3 accounting to include terms treated as lower order in the main text.The expanded forms reduce to Eqs. (5) and (6) when those terms are dropped.Source: A Full Derivations · p. 21
  • It derives the prefill cost model underlying the 56.9 s recompute bound in Section 4.Source: A Full Derivations · p. 21
  • It evaluates the topology model of Eq. (10) per device, with no change to main-text quantities.Source: A Full Derivations · p. 21

A.1 Per-Kernel FLOP and Byte Accounting for One Decode Step

The appendix accounts for decode-step FLOPs and bytes by separating weight reads from attention cache reads, then states omissions and modeling assumptions. It also situates a quality-versus-speedup figure as orientation rather than a protocol-aligned comparison.

A.1 Per-Kernel FLOP and Byte Accounting for One Decode Step

  • Table 8 accounts for per-layer weight matrix-vector products and attention score/value contractions, assuming a single cache read without spilling the score matrix to HBM.The softmax term is omitted in the main text; its omission is 0.59 percent at dh = 128 and the resulting relative error is below 0.6 percent.Source: §A.1 · pp. 21–22
  • Figure 11 plots speedup against normalized retention quality, with a non-dominated frontier reaching 6.6× for composed 2-bit quantization plus eviction.Quality values come from differing task suites and are presented for orientation, not as protocol-aligned comparisons.Source: Figure 11 · p. 22

A.2 Byte Accounting and Kernel Assumptions

  • The byte model counts weight reads once per step and cache reads once per sequence, while omitting new-token cache writes and activation/logit traffic.The omitted cache write is a 1/s correction; activation and logit traffic is at most 0.26 MB per sequence versus 182 GB at W1.Source: §A.2 · p. 22
  • The model assumes dequantization is off the critical path and perfect compute-memory overlap, so step time is bytes divided by bandwidth.Under the stated workloads, the simplified byte equation has relative error below 10^-5.Source: §A.2 · p. 22

A.3 Prefill Cost and the Worked Recompute Bound

At 128k context, prefill work is split nearly evenly between weight matmul and quadratic causal attention, while recomputing a warm-prefix suffix costs more than the section’s suffix-local estimate. The corrected count narrows—but does not remove—the warm-prefix TTFT advantage.

A.3 Prefill Cost and the Worked Recompute Bound

  • Figure 12 scores domains across six capability dimensions from 0 to 5; no domain dominates, and only the composed design space covers the hexagon.Byte reduction and decode speedup reflect Eq. (14) at W2, while lossless domains receive a quality-safety score of five.Source: Figure 12 · p. 23
  • 55% of prefill work at 128k comes from the quadratic attention term; doubling context multiplies recompute time by 3.1×, approaching 4× asymptotically.The 128k calculation totals 3.940 × 10^16 FLOPs, comprising 1.792 × 10^16 weight-matmul FLOPs and 2.148 × 10^16 causal-attention FLOPs.Source: §A.3 · p. 23
  • 1.39 × 10^16 FLOPs, or 20.0 s, is the exact W3 suffix recompute cost, reducing the warm-prefix TTFT advantage from 6.8× to 2.8× versus 56.9 s cold prefill.The 8.4 s estimate retains only suffix-local terms; 6.8× remains a suffix-local lower bound, and the conclusions of Sections 4 and 5 are unchanged.Source: §A.3 · p. 23

A.4 Evaluation of the Topology Model

The topology model yields device-specific effective bandwidth: H100 exposes its aggregate rate, while B200 and MI300X placement choices can limit access to aggregate bandwidth. The appendix states that its constants are reproducible from primary specifications.

A.4 Evaluation of the Topology Model

  • H100 is monolithic, so its effective bandwidth is 3.35 TB/s for any placement.The rate ceiling for Llama-3-70B is defined as effective bandwidth divided by 327,680 bytes per token.Source: §A.4 · pp. 23–24
  • On B200, striped placement reaches the 8.0 TB/s aggregate rate, but pinned placement gives 4.0 TB/s, half of aggregate.The package has two dies with 4.0 TB/s per-die bandwidth and a 10 TB/s NV-HBI fabric; striping is sufficient because the fabric does not bind.Source: §A.4 · p. 23
  • On MI300X, pinned placement yields 0.66 TB/s, eight times below the 5.3 TB/s aggregate; striped bandwidth depends on the undisclosed on-package fabric rate.The main-text striped figures assume the fabric does not bind.Source: §A.4 · p. 24

B Per-Model and Per-Device Constants

  • The appendix makes the model’s constants reproducible from primary specifications and reports byte quantities in decimal units.The configuration procedure takes model, precision, workload, device, and quality-budget inputs and returns a domain configuration.Source: B Per-Model and Per-Device Constants · p. 24, §A.4 · p. 24

B.1 Per-Token KV Bytes of Table 1

Per-token KV storage is determined by model dimensions and precision; at 128k context, the resulting footprints vary substantially across models. MLA’s shared latent representation removes the head-count factor and reduces the constant relative to GQA.

B.1 Per-Token KV Bytes of Table 1

  • At 128k context, KV footprints are 104.9 GB for Llama-2-13B, 16.8 GB for Mixtral 8x7B and Llama-3-8B, 41.9 GB for Llama-3-70B, 66.1 GB for Llama-3.1-405B, and 8.8 GB for DeepSeek-V2.The values follow by multiplying the per-token quantity by 128,000.Source: §B.1 · p. 24

B.2 Parameter Counts Used in the Derivations

The parameter counts used in derived quantities are reconstructed from layerwise counts plus input and output embeddings, with rounding that shifts crossover lengths by less than one percent.

B.2 Parameter Counts Used in the Derivations

  • Parameter count P is reconstructed from the layerwise count plus input and output embeddings at vocabulary size 128,256; rounding shifts s∗ and scap by under 1%.Derived quantities use the rounded counts in the final column of Table 9.Source: §B.2 · p. 24

B.3 Crossover and Capacity Arithmetic

The appendix quantifies traffic and capacity crossovers, hardware rate ceilings, and cache sharing and transfer costs. It also distinguishes analytical, derived, and reported evidence and states the assumptions behind derived values.

B.3 Crossover and Capacity Arithmetic

  • Traffic crossovers are 122.5k, 427.2k, and 1.57M tokens for Llama-3-8B, Llama-3-70B, and Llama-3.1-405B, respectively.Other batch-size entries scale as 1/b.Source: §B.3 · p. 24
  • At the capacity crossover, Llama-3-70B fits to 159k tokens on B200 and MI300X, but its 140 GB weights do not fit on one 80 GB H100.Llama-3.1-405B's 810 GB weights exceed the capacity of all three devices, requiring multi-device placement or weight quantization before context length matters.Source: §B.3 · p. 25

B.4 Ridge Points, Fixed Point, and Rate Ceilings

  • For Llama-3-70B, rate ceilings range from 2.0M tokens/s on pinned MI300X to 24.4M on striped B200.The ridge points are 295 FLOP/B on H100, 281 on B200, and 247 on MI300X; the BF16 GQA intensity fixed point is b† = 8.Source: §B.4 · p. 25

B.5 Sharing and Transfer Arithmetic of Section 4

  • Sharing one 96k prefix across eight 32k suffixes yields σ = 2.91; fetching a 41.9 GB cache takes 0.66 s over PCIe Gen5 x16 or 12 ms over codec-assisted NVLink.The fetch path is 87× to 4,900× faster than the 56.9 s recompute.Source: §B.5 · p. 25

C.1 Notation

  • The notation tables collect model, attention, workload, cost, and hardware symbols, with defining-equation references for workload and hardware symbols.The running example is Llama-3-70B in BF16 on one B200 with striped pages.Source: §C.1 · p. 25, Table 11 · p. 26

C.2 Units and Numerical Conventions

  • Calculations use decimal SI units and token counts; vendor throughput and bandwidth figures are specifications, and derived values are rounded to three significant figures.Unless another precision is stated, s*, scap, and R∞ refer to the BF16 cache with wkv = 2 bytes.Source: §C.2 · p. 26

C.3 Evidence Classes and Mixing Rules

  • The paper separates quantitative claims into analytical closed forms, derived point evaluations assuming perfect kernel efficiency, and within-paper reported values that are not compared across papers.Reported values are confined to Table 7's quality column and the domain discussions.Source: §C.3 · p. 26

C.4 Workload Parameterization and Measurement Assumptions

  • Workloads are parameterized for Llama-3-70B on B200 at 128k context, while derived values assume single-read attention, roofline-bound decode, and 70% peak FLOP utilization for prefill.Dequantization and gather overheads are excluded from speedup columns, with realized overheads discussed in Section 5.Source: §C.4 · pp. 26–27, Table 12 · p. 27

Related papers

Loading 2609.30854v1…