Source-linked AI summary
Characterizing High Bandwidth Flash for LLM Serving
Zack Yu, Chloe Wong, Coleman Hooper, Minjae Lee, Wonjun Kang, Youngjin Cho, Michael W. Mahoney, Yakun Sophia Shao, Kurt Keutzer, Amir Gholami
TL;DR
Growing model weights, KV caches, and reusable agentic context pressure accelerator memory, while HBF adds capacity with different access costs and finite write endurance. The paper evaluates an HBM–HBF–host hierarchy and buffered cache-aware scheduling using trace-driven simulation. On the reported 1× traces, hybrid systems retaining weights in HBM reduce completion time and modeled energy relative to HBM-only serving.
Problem
Agentic serving grows KV state across repeated calls, making memory capacity and retaining reusable prefixes important, while HBF’s read energy, slow writes, and finite endurance complicate its use.
Source: §1 · p. 1Method
The paper evaluates an HBM–HBF–host hierarchy with buffered cache-aware scheduling using a request-level simulator that tracks placement, cache residency, data movement, and execution.
Source: §1 · p. 2Results
On 1× Llama-3.1-405B and GLM-5.2 traces, hybrid systems retaining weights in HBM reduce completion time by 46–56% and modeled energy by 14–24% relative to HBM-only serving.
Source: §6 · p. 9Takeaways & Limitations
For high-throughput serving, HBF’s benefits depend on coordinating placement and scheduling with its access costs; the reported savings mainly come from retaining reusable KV state on-device and amortizing weight reads over larger batches.
Source: §1 · p. 1, §6 · p. 9The study excludes latency-constrained serving and does not model a production HBF controller, complete package power, or wear leveling.
Source: §5.5 · p. 9
Abstract
from arXiv · showhide
Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it increasingly important to retain KV state for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for large language model (LLM) serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and energy efficiency. We introduce an HBM-HBF-host hierarchical storage system and buffered cache-aware scheduling, and use trace-driven simulations to analyze their effects on performance, energy consumption, and HBF write lifetime. Across the evaluated workloads, the fastest HBF-augmented systems reduce completion time by 36.1-87.0% relative to HBM-only systems. Modeled energy savings reach 55.8%, although HBF increases energy consumption on some light workloads. Buffered cache-aware scheduling extends estimated HBF write lifetime from 4.77 to 14.82 years in the evaluated configuration. These results demonstrate the importance of coordinating data placement and scheduling to improve serving efficiency while sustaining a practical HBF write lifetime.
1 Introduction
Agentic LLM serving must balance memory capacity for growing KV state against the value of retaining reusable prefixes. The paper evaluates an HBM–HBF–host hierarchy and buffered cache-aware scheduling for high-throughput workloads.
1 Introduction
- Agentic sessions can reuse completed requests’ KV state, but admitting more requests may displace it and require host reloads or repeated prefill.Long contexts increase both the space needed to retain reusable prefixes and the cost of recovering them.Source: §1 · p. 1
- HBF expands device capacity, but its higher read energy, slower writes, and finite endurance make tier placement and admission decisions consequential.The studied configuration contrasts HBF with HBM on read energy and write speed; repeated writes consume finite program/erase endurance.Source: §1 · p. 1
- The proposed HBM–HBF–host hierarchy places new KV state and reloaded prefixes in HBM, while buffered cache-aware admission reserves room for decode growth and reduces pressure to evict reusable caches.The evaluation targets high-throughput workloads that prioritize workload completion time and energy efficiency and can tolerate higher per-request latency.Source: §1 · p. 2
- 46–56% lower completion time and 14.4–24.1% lower modeled energy than HBM-only baselines on 1× Llama-3.1-405B and GLM-5.2 traces are reported for hybrid systems retaining weights in HBM.The study also evaluates buffered scheduling and trace-dependent performance and energy tradeoffs.Source: §1 · p. 2
2 Related Work
Prior work addresses LLM batching, KV management, agent scheduling, and heterogeneous storage, while flash-based inference explores several placement strategies. This paper focuses on coordinating storage and scheduling decisions under HBF’s access costs and finite endurance.
2 Related Work
- LLM serving systems address batching, KV-memory management, and prefix reuse [10] [36], while FlexGen coordinates GPU, CPU, and disk resources for throughput-oriented inference [25].Agent-program scheduling and KV-cache time-to-live are also explored.Source: §2 · p. 2
- Flash-based inference work spans weight offloading, SSD-backed KV swapping, computation in flash, and vector search [1] [35]; concurrent HBF studies examine KV storage, static expert weights, and endurance [14] [27].The paper positions hierarchical placement and buffered cache-aware scheduling as its approach to reducing cache churn and repeated writes.Source: §2 · p. 2
- Existing simulators model heterogeneous serving, distributed training, or agentic-inference memory, while this request-level simulation studies placement and admission across an HBM–HBF–host hierarchy [3] [30] [31].Its stated outcomes include completion time, energy, and writes.Source: §2 · p. 2
3 Methodology
The methodology evaluates HBF for high-throughput serving using fixed-workload completion time and modeled energy, with a request-level simulator that tracks execution, cache state, and tier movement. It replays agentic traces and models kernels, communication, memory energy, and parallelism.
3.1 Goal of Study
- The study measures fixed-workload completion time and modeled energy without a per-request latency target, and also reports cache-hit fractions, KV writes, and endurance-based lifetime estimates.Comparisons hold workload and accelerator count fixed within each experiment; TPOT exposes latency tradeoffs.Source: §3.1 · p. 2
3.2 Simulation Method
- The simulator advances eligible requests through cache preparation, forward execution, and cache movement while tracking HBM, HBF, host DRAM, and SSD residency.It models decode-driven KV growth, eviction and offload, and reloads before later execution; token placement within each tier follows the specified policy.Source: §3.2 · pp. 2–3
- Kernel latency models are calibrated from H100 and B200 measurements, with separate models for GEMM, prefill and decode attention, and sparse MLA/indexer work.Mixed-batch and HBF performance are modeled extrapolations from measured kernels.Source: §3.2 · p. 3
- Modeled energy sums arithmetic, memory-access, and communication costs without static power, so savings come from fewer operations, less data movement, or lower-energy memory accesses.The simulation also models TP, EP, and data-parallel attention; energy sums activity across ranks, and completion time is the latest replica completion.Source: §3.2 · p. 3
3.3 Simulation Trace
- The trace derives from 24,880 turns in 767 LMCache sessions; replicated 25k- and 100k-turn traces increase request counts without extending conversations.A 5× trace concatenates five copies per session with zero gaps between copies and increases mean input length from 22,558 to 89,125 tokens.Source: §3.3 · pp. 3–4
- GLM sparse-attention placement is estimated from offline selection profiles at a 32,768-token reference context, while expert-copy access estimates use one profiled session for training and another for testing.The sparse-selection model caps selected tokens per query at 2,048 and interpolates across sampled window sizes.Source: §3.3 · p. 4
4 System Overview
The system overview compares shared-site and sequential HBM–HBF architectures, then describes a tiered storage hierarchy and buffered cache-aware admission. The design aims to preserve reusable KV state while managing tier capacity, access costs, and cache movement.
4.1 H3 versus Sandisk’s Shared-Site Design
- H3 retains eight HBM stacks and adds eight HBF stacks behind them, while the shared-site design divides eight stack positions between HBM and HBF; modeled H3 read time is no greater even if reads are serialized.The study simulates both architectures; H3 assumes HBM buffering can fully hide latency from HBF.Source: §4.1 · p. 4, Figure 2 · p. 4
4.2 Hierarchical Storage System
- The hierarchy stages new KV state and reloaded prefixes in HBM, moves older state toward HBF and then host storage, and uses LRU to select eligible cache for demotion or offload.Active-request reference locks exclude state from ordinary LRU eviction; index-first placement demotes eligible attention KV before indices under HBM pressure.Source: §4.2 · pp. 4–5
4.3 Buffered Cache-aware Scheduling
- The admission buffer limits aggregate scheduling pressure and preserves cache for reuse; the authors report that it extends estimated HBF write lifetime to over a decade in both evaluated hybrid configurations.The threshold is checked at admission rather than enforced as a fixed physical partition after every decode token.Source: §4.3 · pp. 5–6
5 Results
The results compare HBF placement and cache-aware scheduling in trace-driven, high-throughput workloads, emphasizing completion time, modeled energy, cache behavior, and estimated write lifetime.
5.1 Experimental Setups
- The GLM traces contain 100k turns at 1× and 500k at 5×; Qwen uses a separate 25k-turn workload.Sessions start at simulation time zero, with later turns following the preceding turn and recorded inter-turn gap; the fixed workload has no external request arrival rate.Source: Table 3 · p. 6, §5.1 · p. 6
5.2 Scheduling and Write Endurance
- On HBM alone, the buffer raises device hits from 6.61% to 56.79%; hybrid device hits include HBF residency, not just HBM-speed access.The buffer also reduces HBM KV writes by more than half relative to unbuffered cache-aware scheduling.Source: §5.2 · pp. 6–7
- With a 10% buffer, estimated HBF lifetime rises from 4.77 to 14.82 years on H3 and from 1.06 to 10.57 years on shared-site 4+4.On H3, the buffer reduces HBF KV writes by 69.33%; on shared-site 4+4, writes fall by 91.96%. Lifetime estimates assume uniform wear, unit write amplification, and 100k P/E cycles.Source: §5.2 · p. 7
5.3 Memory Architecture and Weight Placement
- On the 1× trace, H3 with HBM weights cuts completion time 45.95% for Llama and 56.29% for GLM versus HBM-only serving.These comparisons cover Llama-3.1-405B and GLM-5.2 configurations.Source: Table 5 · p. 7, §5.3 · p. 7, Table 6 · p. 8
- Moving weights to HBF offers little completion-time benefit but raises energy; split TP8 cuts energy 25.97% versus all-HBF TP8 at a 4.79% completion-time cost on 1×.For GLM TP16, moving weights to HBF increases energy by 33.42% above HBM-only. Split TP8 is 2.89% faster but uses 7.58% more energy than TP16 with HBM weights on 1×.Source: §5.3 · p. 7
- Among tested shared-site allocations, 6+2 is best on 1×; on 5×, 4+4 completes 14.25% sooner while 6+2 uses 4.01% less energy.On the extended trace, 6+2 reaches 98.11% device hits, while H3 retains the strongest completion-time results in these comparisons.Source: Table 6 · p. 8, §5.3 · pp. 8–9
5.4 Sources of Energy Savings
- On 1×, H3 with HBM weights saves baseline energy by 24.06% for Llama and 14.41% for GLM.For Llama, avoided prefill saves 20.25 percentage points of baseline energy, including 53.33 million query-token computations; batching cuts weight-read energy by 77.8%. H3 GLM retains 13.09% savings when HBF read energy is repriced from twice to four times HBM read energy, at fixed traffic.Source: Table 7 · p. 9, §5.4 · p. 9
5.5 Scope and Limitations
- The evaluation targets high-throughput completion time and energy, not latency-constrained serving; every tested hybrid configuration has higher mean TPOT than HBM-only on 5×.Kernel fits and layer-class approximation limit extrapolation beyond the profiled hardware.Source: §5.5 · p. 9, §5.3 · p. 9
6 Conclusion
High-bandwidth flash can improve high-throughput LLM serving when placement and scheduling account for its costs. The conclusion reports performance and energy gains alongside longer estimated write lifetime from admission control.
6 Conclusion
- On 1× traces, HBM-weight HBF systems cut completion time 46–56% and modeled energy 14–24% versus HBM-only serving.The evaluated models are Llama-3.1-405B and GLM-5.2.Source: §6 · p. 9
- On the GLM-5.2 5× trace, the HBM-weight configuration achieves a 6.7× speedup and 56% energy savings.Source: §6 · p. 9
- Reserving 10% headroom reduces HBF KV writes by 69% and extends estimated write lifetime from 4.8 to 14.8 years.Source: §6 · p. 9
- The reported gains come mainly from retaining reusable KV state on-device and amortizing weight reads over larger batches.Source: §6 · p. 9