Source-linked AI summary
HBFSim: A Simulator for Exploring HBF in Memory Hierarchies for LLMs
Jinheng Li, Guosheng Wang, Hongchao Du, Shangyu Wu, Qiao Li, Chun Jason Xue
TL;DR
LLM memory demand and flash behavior make HBM+HBF serving performance and lifetime difficult to predict before HBF hardware is available. HBFSim couples a serving frontend with HBM, HBF, and capacity-tier models. Its studies report faster-than-host-DRAM spill service, HBM placement guidance, and HBF lifetime estimates under a published peak load.
Problem
Before HBF hardware is available, designers lack evidence on tier choice, HBM capacity and placement, FTL handling of KV writes, and HBF lifetime.
Source: §1 · p. 1, §2.3 · p. 3Method
Results
HBF serves data outside HBM 11.0–19.0× faster than host DRAM; HBM suits running KV better than caching HBF, and Llama-3.1-70B HBF lifetime is 2.0 to 6.9 years at published peak load.
Source: §1 · p. 2, §7 · p. 10Takeaways & Limitations
HBM should hold running sequences’ KV and HBF read-only weights, with KV moving to HBF only when it must leave HBM.
Source: §7 · p. 10The studies pair an H200 compute profile with a presently nonexistent HBM4/HBF system, so HBF timing and endurance remain assumptions.
Source: §5 · p. 9
Abstract
from arXiv · showhide
High Bandwidth Flash (HBF) has recently emerged as a promising technology that provides terabyte-scale accelerator-local capacity at bandwidths approaching HBM, with a potential cost-per-bit advantage over HBM. However, its NAND foundation brings long latency, out-of-place updates, and finite write endurance directly into the accelerator memory hierarchy, making its effects on performance and lifetime difficult to predict. To study this emerging design space, we present HBFSim, an LLM-aware event-driven simulator for heterogeneous HBM+HBF memory systems that preserves application-level memory semantics while modeling flash-internal behavior. HBFSim models HBM, HBF with host-managed flash translation, and conventional capacity tiers in one simulation driven by a calibrated serving frontend. Using HBFSim, we study decoding of Llama-3.1-70B and Qwen3-235B-A22B under three workloads that differ in their input and output lengths. HBF serves data that does not fit in HBM 11.0--19.0$\times$ faster than host DRAM; HBM is better used to hold the running sequences' KV cache than as a cache for HBF; and at a published peak load of Llama-3.1-70B, HBF lasts 2.0 to 6.9 years, in proportion to the stacks that share the KV writes. The artifact is available at https://github.com/FlashLabAE/HBFSim and will track public HBF device data as it becomes available.
1 INTRODUCTION
HBF offers accelerator-local capacity, but flash latency, management, finite endurance, and unavailable hardware make its effects on LLM serving difficult to predict. HBFSim couples serving and memory models to study these trade-offs and reports faster spill service, placement guidance, and lifetime estimates.
1 INTRODUCTION
- HBF’s capacity promise meets LLM memory demand: weights occupy tens to hundreds of gigabytes, while KV grows with context and concurrency.An 8B model with eight concurrent 128K-context sessions can exhaust a modern GPU’s HBM; long-context 70B-class serving may exceed an eight-GPU node.Source: §1 · p. 1
- Flash latency, out-of-place updates, FTL management, write amplification, and finite endurance complicate HBF placement decisions, with no public hardware available for validation.Read-only weights mainly exercise reads and free HBM for KV, whereas mutable KV in HBF creates program traffic and consumes endurance.Source: §1 · pp. 1–2
- HBF serves data that does not fit in HBM 11.0–19.0× faster than host DRAM.The study also finds that HBM is better used for running sequences’ KV than as an HBF cache, and that HBF lifetime depends on KV writes reaching flash.Source: §1 · p. 2
- HBFSim combines HBM, host-managed-FTL HBF, and conventional capacity tiers in one event system coupled to a calibrated serving frontend.It accepts address-level and production request traces and studies Llama-3.1-70B and Qwen3-235B-A22B under three workloads.Source: §1 · p. 2
2 BACKGROUND
LLM serving combines reusable weights with a KV footprint that grows as requests progress, making memory capacity a constraint. HBF offers high-capacity, HBM-like bandwidth, but retains NAND behavior; HBM+HBF design choices therefore depend on serving and flash-management interactions.
2.1 Memory Demand of LLM Serving
- KV stores per-layer keys and values for processed tokens; decoding reads it instead of recomputing, and retained request blocks can expand its footprint.Decoding also reuses weights across a batch and appends one token’s KV per layer, so it is usually bandwidth-bound; offloaded data incurs the slower tier’s access costs.Source: §2.1 · p. 2
2.2 High Bandwidth Flash
- HBF offers 8–16 times HBM-stack capacity at similar interface bandwidth, but NAND page reads take microseconds and programs take tens of microseconds.The host FTL handles mapping, garbage collection, and wear leveling; host placement and FTL write costs shape traffic and lifetime.Source: §2.2 · pp. 2–3
2.3 Questions for HBM+HBF Systems
- Before HBF hardware exists, designers must assess tier choice, HBM capacity and placement, FTL handling of KV writes, and HBF lifetime.These questions interact through serving: slower memory changes decoding time and batching, which changes memory traffic.Source: §2.3 · p. 3
3 HBFSIM DESIGN
HBFSim models LLM serving and heterogeneous memory in a closed loop, preserving application-level scheduling while representing flash behavior. Its design spans trace-driven serving, placement policies, device models, accelerated simulation, and validation against measured behavior and explicit assumptions.
3 HBFSIM DESIGN
- HBFSim connects a serving frontend, memory policies, and device engine in a feedback loop, so memory delays can reshape batching and tier placement.Its interfaces separate model objects, policy-generated transaction graphs, and device execution; the engine itself contains no LLM-specific logic.Source: §3 · p. 3
3.1 Serving Frontend
- Request-level traces let ServeLoop regenerate weight and KV accesses while replaying continuous batching, chunked prefill, prefix sharing, and preemption.Address-level traces replay individual accesses but cover short runs; generated accesses are checked against captured address streams, with prefill traffic a lower bound.Source: §3.1 · pp. 3–4, §3 · p. 3
3.2 Memory Policies
- Peer mode places each object in HBM or HBF by priority, while tiered mode caches HBF objects in HBM using LRU, FIFO, or decayed-LFU.Dirty KV can be written back eagerly or on eviction, and KV released before eviction is discarded without writing.Source: §3.2 · p. 4
3.3 Device Engine
- The device engine models HBM at pseudo-channel granularity and HBF page operations, host-managed mapping and garbage collection, and HBF mapping metadata competing for HBM.The HBF model includes page-, extent-, and block-level mappings; other modeled tiers include host DRAM, CXL SSDs, and NVMe SSDs.Source: §3.3 · p. 4
3.4 Simulation Speed
- 24.8× faster all-HBM forward passes retain batch completion times within 0.04%; page grouping and lazy FTL recounting also accelerate flash batches.The reported speedups are 1.48× for a Llama-3.1-70B read batch and 1.55× for ten write batches, with bit-identical completion times across 82,449 transactions.Source: §3.4 · pp. 4–5, Figure 3 · p. 5
3.5 Calibration and Validation
- Calibration matches vLLM batching and fits GPU operators and host DRAM copies to measurements, but HBM4/HBF timings remain assumptions and 18 of 30 parameter groups are assumed.The engine also checks accounting on every run and compares HBM scheduling, HBF reads and writes, and garbage collection against reference models.Source: §3.5 · p. 5
4 CASE STUDIES
HBFSim evaluates HBM+HBF configurations for Llama-3.1-70B and Qwen3-235B-A22B across three serving workloads, tracing performance, placement, flash translation, and endurance. The case studies find that HBF is valuable when data spills beyond HBM, while HBM placement and flash read parallelism shape performance and KV writes determine estimated lifetime.
4.1 Setup
- The study models Llama-3.1-70B and Qwen3-235B-A22B across three 8K-token workloads, replaying 32-sequence batches and measuring decode windows at median context.The workloads average 5.2K, 6.3K, and 7.3K context tokens; their decode steps read 51–71 GiB of KV for Llama and 30–42 GiB for Qwen.Source: §4 · p. 5, §4.1 · pp. 5–6
4.2 HBF Versus Other Capacity Tiers
- HBF serves spill 11.0–14.4× faster than host DRAM for Llama-3.1-70B and 18.6–19.0× faster for Qwen3-235B-A22B; off-package tiers win only when freed sites let HBM hold the working set.With eight HBM stacks, both working sets fit and TPOT is 49–55 ms; with two HBM stacks, weights still come from the capacity tier.Source: Figure 4 · p. 6, §4.2 · p. 6
4.3 How Much HBM, and What Should It Hold?
- HBM should prioritize running KV: at 2 HBM + 6 HBF, weights-first takes 4–14% longer for Llama, while Qwen’s decode is dominated by expert-weight reads.For Llama, KV-first takes 69–83 ms versus 75–95 ms for weights-first; Qwen’s KV-first TPOT falls gradually as HBM increases from 1 to 4 stacks.Source: §4.3 · pp. 6–7, Figure 5 · p. 7
- When the active set exceeds HBM, tiered replacement rereads data and generally trails KV-first; decayed LFU’s relative performance varies with stack split and expert popularity.Under skewed expert popularity, decayed LFU reaches 0.78 of KV-first time at α=2 with 4 HBM + 4 HBF, but remains 1.07–1.10× slower at 2 HBM + 6 HBF.Source: §4.3 · pp. 7–8
- At batch 64, only long OUTPUT fits in 6 HBM + 2 HBF; during prefill, write strategies have similar request times, but background draining writes 70–71 GiB to flash.That is 1.2–1.6× the write-back-on-eviction traffic; KV-first decode programs no HBF page, while LRU tiered mode programs 61,000–65,000 pages per Qwen step.Source: §4.3 · p. 8
4.4 Flash Translation and Endurance
- Extent mapping stays within 0.5% of page-level decode and prefill performance, whereas block mapping is 4.2–4.3× slower in decode because it serializes page reads on one plane.Extent mapping uses a 0.9 MiB index for touched pages, compared with 97–152 MiB for page-level mapping.Source: Figure 8 · p. 8, §4.4 · p. 8
- At the published peak load, estimated HBF lifetime is 24–83 months for Llama-3.1-70B and 40–141 months for Qwen3-235B-A22B, in proportion to stacks sharing KV writes.The estimate assumes 100,000 P/E cycles; measured flash bytes erased per written byte are 1.0 within 0.1% for either mapping.Source: §4.4 · pp. 8–9, Table 3 · p. 8
4.5 Which HBF Parameters Matter?
- Flash-array page-sensing rate and planes per die drive decode TPOT; interface speed grade, write parameters, and thermal throttling have little effect in the evaluated read-heavy setting.Doubling page read time to 8 μs raises TPOT by 30–42%, while doubling interface speed grade changes it by 0.1%.Source: Figure 9 · p. 9, §4.5 · p. 9
5 DISCUSSION
The discussion bounds the case studies and identifies assumptions that qualify their performance and lifetime findings, while describing how HBFSim can be extended.
Limitations
- The case studies reproduce only one decoding step, seven tokens per sequence, and one prefill window; they omit garbage collection, so TPOT excludes its interference with reads.The studies model one accelerator, and tensor or pipeline parallelism would give each a share of weights and KV.Source: §5 · p. 9
- HBF timing and endurance remain assumptions because the studies pair an H200 compute profile with HBM4 and HBF, a combination that does not exist today.Lifetime estimates assume a published serving load around the clock and perfect wear leveling.Source: §5 · p. 9
Extending HBFSim
- Users can extend HBFSim through interfaces for memory policies and host mappings, device configurations, and model, trace, or GPU descriptors and profiles.Source: §5 · p. 9
6 RELATED WORK
Related work spans hybrid HBM–flash designs, memory and storage simulators, and LLM serving simulators. HBFSim combines transaction-level flash modeling with LLM serving and a host-side FTL.
HBF and hybrid HBM–flash memory
- H3 and SK hynix evaluate projected hybrid HBM+HBF organizations, FlashAccel [32] co-designs HBF data layout and storage, and Petrucci et al. identify writes as a limit for KV offloading.HBFSim compares such designs on the same workloads and quantifies the write problem through endurance results.Source: §6 · p. 9
Memory and storage simulators
- Ramulator 2.0 and DRAMsim3 model DRAM commands, while MQSim, FEMU, and Cylon model SSDs or CXL SSDs with device-side flash management.HBFSim treats HBF as memory with a host-side FTL whose metadata competes with the application in HBM.Source: §6 · p. 9
LLM serving simulators
7 CONCLUSION
The conclusion presents HBFSim as a way to explore HBM–HBF choices before hardware exists. Its case studies report lower TPOT than host DRAM and NVMe SSDs, recommend keeping running KV in HBM, and identify endurance as a constraint.
Conclusion
- HBF keeps TPOT 11.0–19.0× lower than host DRAM and 86.6–154.1× lower than an NVMe SSD when serving data that does not fit in HBM.Source: §7 · pp. 9–10
- Once a step’s read set exceeds HBM, no conventional replacement policy matches peer mode with KV first; a cache wins only when that set fits with room to spare.HBM should hold running sequences’ KV and HBF the read-only weights, with KV moving to HBF only when it must leave HBM.Source: §7 · p. 10
- An extent index preserves flash-array parallelism with a fraction of a page table’s memory.Source: §7 · p. 10
- At a published peak load of Llama-3.1-70B, projected HBF lifetime is 2.0 to 6.9 years, in proportion to the stacks sharing KV writes.Source: §7 · p. 10