PaperlayerView exact revision on arXiv

Source-linked AI summary

HBFlex: A Flexible Memory System for Bridging Fine-Grained LLM States and Coarse-Grained HBF Parallel Execution

Shuzhang Zhong, Weikai Xu, Yifan Zhou, Tongbin Zhao, Tenghao Zhao, Yifei Kang, Cunyin Chang, Shu Li, Guangyu Sun, Meng Li

arXiv:2609.18675v1cs.AR

TL;DR

Growing model weights and KV caches increase accelerator memory demands, while retaining HBM in hybrid designs limits HBF resources under a fixed package budget. HBFlex coordinates KV placement, reads, writes, and reclamation in a full-HBF system. Across evaluated configurations, it achieves average throughput speedups of up to 1.58× over FlashAccel and 3.30× over H3.

  • Problem

    Growing model weights and KV caches pressure accelerator memory capacity, while retaining HBM under a fixed package budget reduces the HBF resources available.

    Source: §1 · p. 1

  • Method

    HBFlex combines HBF-aware placement and attention scheduling, window-aware writeback, and lifetime-aware reclamation, supported by base-die SRAM and co-located prefill and decode.

    Source: §1 · p. 2, §5.1 · p. 7

  • Results

    Average throughput speedups reach 1.58× over FlashAccel and 3.30× over H3 across evaluated configurations.

    Source: §1 · p. 3

  • Takeaways & Limitations

    HBFlex coordinates KV reads, writes, and erases while preserving fine-grained prefix reuse in a full-HBF organization.

    Source: §7 · p. 12

    Under the stated workload assumptions, raw KV writes consume 15.3% of the estimated endurance budget for DeepSeek-V3 and 41.5% for Qwen3.

    Source: §3.3 · p. 5

Abstract

from arXiv · show

Large language models (LLMs) require increasing memory capacity to accommodate growing model weights and KV caches. High-Bandwidth Flash (HBF) offers high memory density and aggregate read bandwidth through massive plane-level parallelism, making it an attractive option for LLM serving. However, serving LLMs entirely from HBF introduces three challenges: fine-grained KV reads create placement and access imbalance, incremental writes interfere with foreground reads, and mixed KV lifetimes amplify garbage collection. Hybrid HBM/HBF designs retain HBM to support dynamic KV management, but this allocation reduces the HBF resources available under a fixed packaging budget, limiting aggregate HBF bandwidth. We present HBFlex, a full-HBF memory system with coordinated optimizations for KV reads, writes, and reclamation. HBFlex balances KV placement and attention accesses to improve plane utilization. It aggregates incremental updates and schedules writeback within sufficiently long compute windows to reduce write--read interference. It also combines lifetime-guided block packing with deferred reclamation to reduce valid-page migration. We evaluate HBFlex through trace-driven simulation across different configurations. HBFlex achieves average throughput speedups of up to 1.58$\times$ over FlashAccel and 3.30$\times$ over H3, benefiting from higher HBF bandwidth and more efficient management of dynamic KV-cache reads, writes, and erases.

1 Introduction

HBF can expand accelerator-local memory capacity, but retaining HBM under a fixed package budget limits the HBF resources and plane-level bandwidth available. HBFlex addresses the resulting full-HBF challenges with coordinated KV management and reports throughput speedups over hybrid baselines.

1 Introduction

  • Under a fixed package budget, retaining HBM limits HBF capacity and read parallelism, while full-HBF serving requires coordinating KV reads, programs, and erases.Fine-grained reads can leave bandwidth unused, incremental updates can block foreground reads, and mixed KV lifetimes can increase garbage-collection data movement.Source: §1 · pp. 1–2
  • Request completion and retained-KV reactivation can disrupt balanced placement, while fine-grained attention reads can concentrate pages on a few planes.Such collisions require additional read waves while other planes remain idle.Source: §1 · p. 2
  • Incremental KV programs occupy target planes for about 75 μs and cannot be preempted, so write aggregation alone cannot prevent interference with foreground reads.Deferring writes eventually creates buffer pressure, making write aggregation and program timing both necessary.Source: §1 · p. 2
  • HBFlex combines HBF-aware placement and scheduling, window-aware writeback, and lifetime-aware reclamation in a full-HBF system with base-die SRAM and co-located prefill and decode.Its design preserves a fine-grained logical interface while coordinating reads, writes, and erases.Source: §1 · p. 2
  • Average throughput speedups reach 1.58× over FlashAccel and 3.30× over H3 across evaluated configurations.The reported gains benefit from higher HBF bandwidth and more efficient KV-cache management.Source: §1 · p. 3

2 Background

HBF exposes aggregate bandwidth through concurrent plane operations, while LLM attention maps logically paged KV state to scattered physical reads. Hybrid HBM/HBF designs retain DRAM for dynamic or latency-sensitive accesses, trading away some HBF resources under fixed package constraints.

2.1 High-Bandwidth Flash Basics

  • HBF’s aggregate bandwidth depends on concurrent plane operations, balanced access loads, and sufficient outstanding requests.Accesses to one plane serialize, so the most heavily loaded plane determines completion time while others may idle.Source: §2.1 · p. 3

2.2 LLM Workload

  • PagedAttention maps logical KV pages to independently allocated physical pages, and attention CTAs fetch request-dependent KV chunks through those mappings.The resulting reads can be scattered across physical locations rather than contiguous.Source: §2.2 · p. 3

2.3 HBF for LLM Serving

  • Hybrid HBM/HBF architectures place weights and read-only shared KV in HBF while retaining dynamic KV in HBM, limiting HBF resources under fixed package or stack budgets.Managing dynamic KV directly in HBF could reduce dependence on HBM but must reconcile KV access and update patterns with page-level operations and block-level reclamation.Source: §2.3 · p. 3

3 Motivation

HBF’s plane-parallel bandwidth depends on balanced accesses, but dynamic KV behavior complicates placement, read, write, and erase management. The section characterizes three challenges in mapping fine-grained KV activity onto HBF’s coarse-grained operations.

3 Motivation

  • HBF’s coarse-grained, plane-parallel operations must be reconciled with fine-grained KV accesses and dynamic updates to realize its bandwidth.The analysis uses a six-stack configuration with 3,072 planes and configured read bandwidth of 2.93 TB/s across six stacks.Source: §3 · p. 4

3.1 Challenge 1: Fine-Grained KV Cache Creates Read Imbalance

  • Request completions and reactivations can disrupt balanced KV placement, while fine-grained attention reads can collide on planes even with balanced placement.Trace analysis finds repeated placement-imbalance and hot-plane-fraction spikes; a 12-MiB read can require extra read waves when pages collide.Source: §3.1 · p. 4
  • Larger logical pages reduce plane occupancy for a 12-MiB read but increase prefill amplification by discarding reusable page tails.Tying logical reuse granularity to physical access granularity trades fine-grained reuse for plane-level read parallelism.Source: §3.1 · p. 4, Figure 7 · p. 5

3.2 Challenge 2: Incremental KV Updates Create Write–Read Interference

  • Incremental KV programs occupy a target plane for about 75 μs and cannot be preempted, so short compute gaps can leave foreground reads stalled.Effective writeback requires both aggregating updates and scheduling programs within sufficiently long compute windows.Source: §3.2 · p. 5

3.3 Challenge 3: Interleaved KV Lifetimes Create Erase Amplification

  • Mixed request lifetimes require valid pages to migrate before block erasure; DeepSeek-V4-Pro averages approximately 30× GC write amplification.Request-isolated blocks reduce lifetime mixing but restrict plane striping and can create load imbalance.Source: §3.3 · pp. 5–6, Figure 10 · p. 5

4 HBFlex Architecture

HBFlex uses a full-HBF package while preserving the serving runtime’s fine-grained logical KV interface. Its architecture coordinates placement, writeback, and reclamation across GPU, base-die, and NAND components.

4.1 Design Considerations

  • HBFlex dedicates the package’s memory budget to HBF rather than retaining HBM, maximizing KV-cache capacity and preserving plane-level parallelism.The design rationale identifies garbage collection, rather than raw KV write volume, as the dominant endurance concern under its workload analysis.Source: §4.1 · p. 6
  • Co-located prefill and decode allow decode KV updates to be buffered locally and drained during prefill compute windows.The design prefetches weights of consecutive operations into base-die SRAM to create longer intervals without intervening weight reads.Source: §4.1 · p. 6
  • Base-die SRAM is shared across read prefetching, decode-KV writeback, hot-plane offload, and garbage-collection migration.HBFlex provisions 40 MB per stack, or 240 MB across six stacks, and forces writeback as shared occupancy approaches capacity.Source: §4.1 · p. 6

4.2 Architecture Overview

  • The GPU issues attention requests through D2D to a base die that coordinates NAND planes, while runtime modules manage logical pages, read scheduling, and write/erase decisions.HBFlex distinguishes logical KV pages, physical stripes, attention work tiles, and erase blocks so reuse, placement, scheduling, and reclamation can be optimized independently.Source: §4.2 · p. 7

5 HBFlex System Design

HBFlex coordinates KV placement, attention scheduling, writeback, and reclamation to make dynamic KV management fit HBF’s physical constraints. Its mechanisms balance plane use, exploit compute windows for writes, and reduce valid-page migration.

5.1 Design Overview

  • The design coordinates three mechanisms—placement and scheduling, window-aware writeback, and lifetime-aware reclamation—to manage dynamic KV while preserving fine-grained logical reuse.The mechanisms target read imbalance, write–read interference, and erase amplification, respectively.Source: §5.1 · p. 7

5.2 HBF-Aware KV Placement and Scheduling

  • Physical page striping and shared-prefix replicas support plane-balanced placement, while runtime redirection and selective SRAM offload balance the active KV working set.Placement-aware chunk construction and CTA scheduling then favors chunks covering more distinct planes, with query locality as a tiebreaker.Source: §5.2 · pp. 7–8

5.3 Window-Aware KV Writeback

  • HBFlex buffers decode KV updates in base-die SRAM and drains them during sufficiently long prefill windows, aggregating writes while reducing interference with foreground reads.Weight prefetching also consolidates computation into longer intervals; when suitable windows are sparse, forced writeback bounds buffer use and temporarily stalls decoding.Source: §5.3 · p. 8, Figure 13 · p. 8

5.4 Lifetime-Aware KV Reclamation

  • HBFlex combines lifetime-guided block packing with deferred reclamation, grouping pages likely to expire together and allowing more pages to become invalid before GC.Cross-layer pages from the same request have related lifetimes; root-request start times also help estimate similarity across request series. Eviction begins at 90% memory utilization, while GC begins at 95%.Source: §5.4 · pp. 8–9

6 Experimental Evaluation

Across evaluated serving configurations, HBFlex improves latency and throughput over the compared systems, with throughput gains increasing at higher concurrency. Trace-driven experiments and component analyses examine its system-level performance and individual mechanisms.

6.1 Experimental Setup

  • The evaluation covers four models, four Alibaba Bailian production traces, and SWE-bench traces at varying task concurrency, using normalized package-area comparisons for cascaded designs.The full-HBF configuration uses six HBF stacks per GPU; cascaded comparisons allocate one quarter of the memory budget to HBM and three quarters to HBF.Source: §6.1 · p. 9

6.2 End-to-End Performance

  • HBFlex achieves TTFT speedups of 1.20× over FA-CLI and 1.33× over FA-CSI, with TPOT speedups of 1.21× and 1.35×, respectively.Against H3, average speedups reach 22.65× in TTFT and 1.89× in TPOT across four Bailian traces and four models.Source: §6.2 · p. 9, Figure 15 · p. 10
  • At the highest SWE-bench concurrency, average throughput speedups reach 1.48× over FA-CLI, 1.58× over FA-CSI, and 3.30× over H3.Across four models, the average speedup over H3 rises from 1.56× at the lowest plotted concurrency to 3.30× at the highest.Source: §6.2 · pp. 9–10, Figure 16 · p. 10

6.3 Ablation Study

  • Read-focused placement and runtime balancing provide most of the cumulative throughput improvement, reaching 4.08× for DeepSeek-V3 and 4.25× for Qwen3-235B over Naive HBF.Writeback adds approximately 0.02× normalized speedup, while reclamation adds approximately 0.05×, bringing overall speedups to 4.15× and 4.32×.Source: Table 2 · p. 10, §6.3 · p. 10

6.4 Performance Analysis

  • Read-wave amplification falls from 5.97× to 3.95× with plane-balanced placement and to 2.25× with runtime balancing, an average reduction of 62.21% versus random placement.Across four Bailian traces and four SWE-bench concurrency levels, write-induced stall falls from 4.54 ms with immediate writeback to approximately 0.003 μs with window-aware writeback.Source: Figure 17 · p. 11, §6.4 · p. 11, Figure 18 · p. 11
  • Average WAF decreases from 11.98× in the baseline to 1.24× with lifetime-guided packing and 1.04× with deferred reclamation, an 89.39% average per-scenario reduction.The complete scheme reduces copying and associated erases, primarily benefiting endurance.Source: Figure 19 · p. 11, §6.4 · p. 11

6.5 Hot-Plane Offload and SRAM Usage

  • Selecting more planes for offload reduces load imbalance but increases SRAM use; beyond 10%, load curves flatten while SRAM use continues to grow.At 10%, offload uses approximately 6–12 MB per stack; at 20%, it uses approximately 14–35 MB per stack. The evaluation selects 10% as the balance point.Source: Figure 20 · p. 11, §6.5 · pp. 11–12

7 Conclusion

HBFlex coordinates KV management in a full-HBF organization to address placement and access imbalance, write–read interference, and GC amplification. Across evaluated configurations, it achieves average throughput speedups of up to 1.58× over FlashAccel and 3.30× over H3.

7 Conclusion

  • HBFlex combines HBF-aware placement and attention scheduling, window-aware writeback, and lifetime-aware reclamation to coordinate KV reads, writes, and erases.Base-die SRAM and co-located prefill and decode support these mechanisms while preserving fine-grained prefix reuse.Source: §7 · p. 12
  • Across evaluated configurations, HBFlex achieves average throughput speedups of up to 1.58× over FlashAccel and 3.30× over H3.The paper attributes the gains to higher HBF bandwidth and more efficient KV-cache management.Source: §7 · p. 12
  • The results support coordinating KV management to exploit full-HBF memory resources.Source: §7 · p. 12

Related papers

Loading 2609.18675v1…