PaperlayerView exact revision on arXiv

Source-linked AI summary

HBFSim: Fast and Faithful Simulation of High-Bandwidth Flash Under Real GPU Execution

Yanpeng Hu, Yiwei Yang, Yuanwu Zhu, Yusheng Zheng, Andi Quinn, Wei Zhang

arXiv:2609.09800v3cs.ARcs.ET

TL;DR

Evaluating HBF for LLMs requires realistic evidence about its effects on real workloads, including preserving GPU concurrency that hides access latency. HBFSim injects modeled HBF service into workloads running on real GPUs, and its reported evaluations include end-to-end execution and thermal-model validation. The paper’s power observations are board-level rather than memory-chip measurements.

  • Problem

    Evaluating HBF requires modeling its program-visible delays on real workloads without sacrificing GPU concurrency that can hide access latency.

    Source: §3 · p. 4

  • Method

    HBFSim rewrites PTX to route registered-range accesses into the simulator and places modeled waits at consumption while supporting capacity through a bounded HBM frame cache.

    Source: §4.2 · p. 5, §4.5 · p. 7, §9 · p. 13

  • Results

    Qwen3-30B-A3B ran end to end under vLLM, with 44.469 s of generation producing the eight requested output tokens; the reduced thermal model agreed with the full-order model to 0.0308 K across eleven outputs.

    Source: §6.2 · p. 9, §6.3 · p. 11

  • Takeaways & Limitations

    The coupled package model represents temperature-dependent service rate and refresh traffic, which the authors use to examine constraints on future HBF accelerator designs.

    Source: §9 · p. 13

    The reported power observations are board-level, not memory-chip temperatures, and the card exposed no memory-temperature readings in 3,877 samples.

    Source: §6.3 · p. 11

Abstract

from arXiv · show

High-Bandwidth Flash (HBF) places high-capacity NAND beside HBM to relieve the memory-capacity bottleneck of LLM inference, yet its system-level behavior cannot be evaluated before hardware becomes available. Cycle-level GPU simulators are too slow for production-scale models. Trace replay has a further shortcoming: it cannot capture the allocation, migration, and execution changes induced by different HBM-HBF configurations. Our key insight is that HBF need not be evaluated by simulating the GPU: only the program-visible effects of HBF need to be modeled. And only a real LLM workload running on real hardware can answer the arguments about HBF. Hence the modeled service has to be injected into that running program, and the injection must not destroy the GPU concurrency that would hide the original I/O latency. We present HBFSim, an open-source HBF simulator that executes LLM workloads on a real GPU while modeling HBF timing, thermal, and other behaviors online. HBFSim rewrites the PTX of the workload's kernels and routes accesses inside a registered address range into the HBF simulator. It supports asynchronous TMA transfers and capacities beyond physical GPU memory. HBFSim leaves the model's run unaffected across ordinary-memory, TMA, and capacity-mode tests. The delay it injects matches the delay requested to within 0.152%. We also design a coupled thermal module that puts HBF, HBM, and the GPU in one advanced package, which is important for answering how severe the hot throttling problem becomes after HBF runs for a long time. Experiments with Qwen3-30B show how package heating, HBM-HBF allocation, and shared MoE demand jointly constrain the design space of future HBF accelerators. The source code of HBFSim is available at https://github.com/SlugLab/hbfsim/.

1 Introduction

LLM inference strains HBM capacity and cost, while HBF offers an in-package NAND tier whose behavior remains contested before hardware is available. HBFSim addresses the evaluation gap by routing a real workload’s registered-range accesses into a simulator while the workload runs on a real GPU.

1 Introduction

  • LLM inference is increasingly constrained by memory bandwidth and capacity, while HBM capacity lags model parameter counts and faces high cost and scarce supply.Table 1 compares four accelerator I/O paths, but the supplied caption does not provide cell values.Source: §1 · p. 1, Table 1 · p. 2
  • HBF adds an in-package NAND tier below HBM, with projected capacity and bandwidth advantages but NAND’s high write cost and microsecond-scale read latency.Its intended fit is data written once and read repeatedly, such as model weights and shared KV-cache prefixes.Source: §1 · p. 1
  • HBF’s potential to ease HBM’s capacity problem is disputed, and designers face choices before the first inference devices are expected to reach customers as samples in 2027.Supporters expect HBF to take over much of HBM’s capacity problem; critics argue NAND’s limits will hold it back.Source: §1 · pp. 1–2
  • Cycle-level GPU simulation is too slow for production-scale LLM runs, while static trace replay cannot capture configuration-driven changes in allocation, migration, and execution.Those limits prevent the approaches from quickly exploring HBF configurations across workloads.Source: §1 · p. 2
  • HBFSim rewrites workload PTX and routes registered-range accesses into the simulator while the LLM executes on a real GPU.Qwen3-30B emits token identifiers identical to the baseline run under injected HBF timing.Source: §1 · p. 2

2 Background

NAND’s limited endurance and slower reads are offset in LLM inference by read-heavy data and predictable execution that permits prefetching. These characteristics motivate studying HBF as a lower-cost memory tier alongside HBM.

2 Background

  • HBF’s NAND medium has limited program/erase endurance and estimated page-read latency of about 20 µs, versus about 200 ns for a GPU global-memory access.The workload must therefore avoid frequent writes and hide latency when data is needed.Source: §2 · p. 2
  • Model weights and shared KV-cache prefixes are read far more often than written; read-only traffic is estimated at 40 to 50 percent of Kimi K3 traffic with a 1M-token context.The paper notes this calculation is not expanded because its focus is HBF simulation.Source: §2 · pp. 2–3
  • Transformer decoding follows a predictable layer order, leaving time to prefetch the next layer’s weights before computation consumes them.On an H100 SXM5, computing one layer leaves about 130.2 µs, or 6.5 times the assumed HBF page-read latency, to fetch the next layer’s weights.Source: §2 · p. 3
  • Because HBF combines lower price and large capacity with bandwidth close to HBM in theory, the authors set out to build a fast, accurate, open-source simulator to study its behavior.Source: §2 · p. 3

3 Motivation

The authors argue that neither cycle-level GPU simulation nor static trace replay can efficiently answer HBF design questions. They motivate running workloads on real GPUs while modeling HBF effects without disrupting asynchronous execution or omitting thermal behavior.

3 Motivation

  • Cycle-level GPU simulators are too slow for acceptable LLM inference runs, and HBF timing parameters are not published in the specification.Table 2 reports simulation time and instruction rate for cycle-level GPU simulators.Source: §3 · p. 3, Table 2 · p. 3
  • A replayed trace is both slow and fixed to one HBM-to-HBF split, so it misses configuration-dependent tensor residency, KV-cache partitioning, and migration kernels.Consequently, one trace cannot efficiently find an optimal configuration across workloads.Source: §3 · p. 3
  • Running a real workload on a real GPU preserves the hardware computation that may hide HBF access latency, but leaves no simulator clock or model to modify.Source: §3 · p. 3
  • Because HBF accesses occur inside the package, conventional host-side or link interception points do not capture them, leaving program-code instrumentation as the remaining route.PTX rewriting is proposed as that interception point, though PTX does not expose the backend schedule and physical-register dependencies.Source: §3 · pp. 3–4
  • A wait at the memory instruction can block a warp that real hardware would continue issuing, while the delay should be paid only when a later instruction consumes the returned value.TMA adds difficulty because hardware determines request expansion, L2 movement, and multicast behavior below PTX.Source: §3 · pp. 3–4
  • HBF’s sustained rate depends on package temperature, and coupled heating can throttle it below nominal bandwidth while refresh traffic also competes with workload reads.The proposed architecture assigns the three stated challenges to design components D1–D5.Source: §3 · p. 4

4 Design

HBFSim closes a feedback loop between a running GPU workload and modeled HBF, routing registered address accesses through online media timing while sharing thermal state and refresh plans.

4.1 D1: An Explicit and Fail-Closed Boundary

  • D1 explicitly registers HBF ranges, classifies effective addresses dynamically, and partitions TMA tiles bytewise across HBM, HBF, and out-of-bounds regions.The PTX pass does not cover precompiled cubins; exact runs are refused when declared access coverage is incomplete.Source: §4.1 · p. 5

4.2 D2: Preserving GPU Asynchrony

  • D2 separates memory issue from consumption: accesses continue after issue, while the first dependent use waits for modeled completion.TMA barriers require both native and modeled completion.Source: §4.2 · p. 5

4.3 D3: A Closed-Loop Thermal Controller

  • D3 couples HBF, HBM, and GPU in one package thermal model, using media-event energy and GPU power to estimate hotspot temperature and govern throttling.A reduced-order model returns package temperature to request service; the controller uses four hysteretic states and snapshots service before queue reservation.Source: §4.3 · pp. 5–7

4.4 D4: Retention, Refresh, and Wear as Injected Latency

  • D4 makes retention and refresh affect the live workload: temperature-driven deadlines trigger refresh writes that contend with foreground reads and increase injected delay.Eligible blocks produce deterministic page-aligned read–rewrite pairs; refresh debt is claimed in quanta during application admission.Source: §4.4 · p. 7

4.5 D5: Capacity

  • D5 models capacity ranges with a bounded HBM frame cache backed by a sparse host file, writing dirty evictions back and checking workload checksums.Refresh uses the backing abstraction without changing payload contents.Source: §4.5 · p. 7

4.6 D6: Speculative Prefetcher

  • D6 predicts MoE accesses from served-page faults; on the synthetic stream it raised accesses served without media reads from 17.97% to 89.16%.Media reads an access still had to wait for fell 86.79%, from 840 to 111. The paper claims no run-time performance benefit from D6.Source: §4.6 · pp. 7–8

5 Implementation

The implementation combines PTX rewriting and reused simulator components with explicit build records, host–device control, and failure-aware run reporting.

5 Implementation

  • The implementation adds range registration, PTX rewriting, shared control, calibrated timing, thermal and retention models, and capacity-mode storage around reused components.It comprises 9,302 lines of C, C++ and CUDA, plus 3,250 lines added for measured calibration; four dependencies are pinned by commit and build records are checked at launch.Source: §5 · p. 8
  • A sealed, fixed-size shared control region holds request and completion slots, with each caller's reservation sequence serving as its completion ticket.Fixed offsets remain valid because the region cannot change size during a run.Source: §5 · p. 8
  • The daemon's first heartbeat marks readiness only after initialization; subsequent heartbeat checks detect a stalled daemon without comparing host and GPU clock origins.A run whose heartbeat stops ends rather than continuing untimed.Source: §5 · p. 8
  • HBFSim reports modeled delay, observed wait, host service time, and overhead separately so simulator software cost is not mistaken for device latency.The four counters are modeled_ns, wall_ns, service_ns, and overhead_ns.Source: §5 · p. 8
  • Failures terminate the run with terminal completions posted to every admitted request, and the summary records state, telemetry, temperature, traffic, wear, and terminal status.Lifetime is reported as a sensitivity sweep, not a measured lifetime for unavailable HBF hardware.Source: §5 · p. 8

6 Evaluation

The evaluation runs Qwen3-30B-A3B on an RTX PRO 6000 Blackwell and tests execution fidelity, timing injection, package heating, and HBM–HBF capacity and demand. Results include timing-injection accuracy, modeled thermal limits, and capacity and routing trade-offs.

6.2 EQ1: End-to-End Execution and Timing Fidelity

  • Across 42 checks, instrumentation preserves values, effective addresses, tier assignments, and synchronization; 128 checksums also match baseline across the 110 GiB logical range.Qwen3-30B-A3B completes end to end and produces the requested eight output tokens.Source: §6.2 · p. 9
  • The reported 798,784.0 ns increment has 0.152% error; this measures injection fidelity, not absolute HBF latency.It compares 1,000 µs and 200 µs injections; no physical HBF latency was measured.Source: §6.2 · p. 9, Figure 2 · p. 10

6.3 EQ2: Thermal Coupling

  • At 86.6 W, the modeled compute die reaches its 90 °C limit before HBM at 121.4 W or HBF at 105.9 W; the eight HBF layers span 44.4 K.Figure 3 shows modeled predictions from a reconstructed thermal operator, not measurements of a running device.Source: §6.3 · p. 10, Figure 3 · p. 10
  • Across seven GPU compute phases, board power spans 314.988–431.370 W while sensor temperature stays about 46–52 °C; the 383.649 GB/s observation is GDDR7 traffic, not HBF service.The card exposes no memory-chip temperature, and the board-power difference is not isolated GDDR chip power.Source: §6.3 · pp. 10–11, Figure 4 · p. 11
  • After one-sample alignment, the 224-state model matches all eleven full-order outputs within 0.0308 K; five thermal policies sustain 384.0 GB/s without throttling across tested horizons.The thermal-model comparison is against a full-order model, not silicon.Source: §6.3 · pp. 10–11

6.4 EQ3: Capacity and Demand

  • Under an 861.1 mm² package-area budget, installed capacity rises from 216 GiB at layout A to 1,572 GiB at C, while HBM-resident KV capacity falls from ∼1.72M tokens at A to ∼0.54M at B.The token figures assume the complete 54.000 GiB expert set is already resident in HBM.Source: §6.4 · p. 11, Figure 5 · p. 11
  • In the Qwen3-30B-A3B routing trace, deduplicated demand grows 2.487× at batch 8 and 3.584× at batch 16, versus routing decisions growing 8× and 16×.These are event counts from three measured batch sizes, not media bytes; prefetching timing benefit was not measured.Source: §6.4 · p. 12, Figure 6 · p. 12

7 Related Work

HBFSim differs from host-side flash simulators by charging modeled flash time to GPU-kernel accesses, and from GPU simulation by instrumenting a real running workload. Related work also shows that pre-silicon full-system models can rely on extrapolated neighboring-standard parameters.

Storage and flash simulators

  • MQSim and SimpleSSD [15] model flash with the host block-device stack, whereas HBFSim reuses MQSim’s media model and charges page-read time to a GPU kernel executing on the GPU.HBFSim advances MQSim’s event queue on demand while the workload runs and intercepts at the GPU instruction path.Source: §7 · p. 12

GPU simulation and GPU instrumentation

  • GPGPU-Sim and Accel-Sim model a GPU, while eGPU instruments a running kernel; HBFSim applies that route to timing and capacity for a device not yet available.SwarmIO [19] emulates GPU-thread reads over NVMe and PCIe, unlike HBF accesses issued by GPU load/store units inside the package.Source: §7 · p. 12

Simulating a device that cannot be obtained

  • Cylon, CXLMemSim, and FlashAccel [41] differ from HBFSim in device tier, observation and delay path, and whether temperature feeds back into reliability behavior.HBFSim models in-package flash read at 64 bytes and written at 4 KiB, observes accesses in rewritten GPU code, and includes the temperature-to-reliability loop specified for HBF.Source: §7 · p. 12

Full-system simulation before the silicon exists

  • Before HBM silicon existed, gem5’s HBM configurations used timing extrapolated from neighboring standards, and its HBM-specific controller arrived six years after HBM2 shipped.The HBM configurations were added in September 2015; the controller with pseudo channels arrived in May 2022, after HBM2 shipped with the P100.Source: §7 · pp. 12–13

8 Discussion

The discussion separates host-paging timing from flash-media timing and describes the coverage and parameter boundaries of the evaluation. Its thermal inputs are measured on one GPU and SSD, while retention results depend on the assumed activation energy.

Timing sources

  • The calibrated end-to-end vmem curve includes page faults, driver handling, page-cache insertion, and scanning, while the online MQSim path models media service without those host-paging steps.Random, reverse, repeated, and cross-operation accesses use the single-page cost; only sequential bursts use the nonlinear curve, which is extrapolated beyond 512 pages.Source: §8 · p. 13

Coverage

  • Coverage depends on each run: already-built kernels are covered at whole-program level, while extending coverage depends on registering ranges and the rewriting layer.For covered accesses, modeled time enters end-to-end time and the output remains unchanged.Source: §8 · p. 13

Hardware parameters and thermal measurements

  • The thermal measurements use one GPU and one Dell CD8P SSD, and simulated-warning crossings at 30 s and 42 s are extrapolated rather than observed on physical devices.The three synthetic timing profiles share 1 TiB capacity and span 128–1,000 GB/s bandwidth; the specification’s approximately 3.0 TB/s and 512 GB per stack can be reflected by editing a data file.Source: §8 · p. 13

Retention

  • Retention deadlines and resulting refresh traffic depend on activation energy, so the evaluation sweeps that parameter rather than reporting one lifetime.The Arrhenius model is anchored at Ea = 1.04 eV, measured for 3D NAND after 10,000 program/erase cycles.Source: §8 · p. 13

9 Conclusion

HBFSim runs a real workload on a real GPU while modeling HBF accesses, capacity, and coupled package temperature. The conclusion reports unchanged outputs, close delay injection, and design trade-offs across thermal, capacity, and MoE-demand experiments.

Core approach

  • HBFSim rewrites PTX to route registered-range accesses into the simulator on a real GPU, supporting asynchronous TMA transfers and capacities beyond physical GPU memory.The modeled wait occurs at the instruction that reads the returned value, and timing is calibrated from real-hardware measurements.Source: §9 · p. 13

Execution and timing fidelity

  • The model’s run is unaffected in ordinary-memory, TMA, and capacity-mode tests, and injected delay matches the requested delay within 0.152%.A run spanning 110 GiB through a 2 GiB page cache returns a checksum identical to a native run’s.Source: §9 · p. 13

Thermal coupling

  • A coupled thermal module puts HBF, HBM, and the GPU in one package, where junction temperature changes service rate and retention-driven refresh traffic.Across 600-, 3,600-, and 86,400-second horizons, five policies sustain 384.0 GB/s without throttling; within that envelope, the modeled compute die reaches its thermal limit at 86.6 W, not HBF.Source: §9 · p. 13

Capacity and placement

  • Under one package-area budget, installed capacity rises from 216 GiB in the all-HBM layout to 1,572 GiB in the most HBF-heavy layout while HBM-held key-value capacity falls.The conclusion presents these layouts as a capacity trade-off rather than an unqualified increase in usable HBM state.Source: §9 · p. 13

MoE demand

  • In the captured Qwen3-30B routing trace, deduplicated expert demand reaches 2.487 and 3.584 times batch-one demand at batches 8 and 16, while routing decisions grow linearly.These demand patterns contribute to the design constraints explored alongside package heating and HBM-HBF allocation.Source: §9 · p. 13

Implication

  • Before HBF parts sample, HBFSim lets inference-system designers measure capacity or placement decisions under a real workload rather than assume their effects.This conclusion is scoped to decisions the simulator can evaluate with its modeled workload and device behavior.Source: §9 · p. 13
Loading 2609.09800v3…