PaperlayerView exact revision on arXiv

Source-linked AI summary

Enabling High-Bandwidth Flash for Generative Recommendation Serving with Write-Aware KV Cache Policy

Danni Peng, Kai Wu, Tianyu Zuo, Pengfei Xia, Hui Zang

arXiv:2609.07175v1cs.AR

TL;DR

GR’s growing KV-cache capacity and bandwidth needs challenge existing memory systems. The paper evaluates admission-controlled LRU-K and models its effects on serving, writes, and HBF lifetime. HBF-based systems achieve 3.8–4.7× higher throughput than HBM-only systems, while LRU-10 extends lifetime from about one year to over six years.

  • Problem

    GR’s growing KV-cache capacity and read-bandwidth requirements expose a mismatch between HBM’s capacity and CPU memory’s bandwidth, while conventional LRU writes full caches on misses.

    Source: §I · p. 1

  • Method

    The paper evaluates admission-controlled LRU-K and develops an analytical model of GR serving performance, KV-cache write traffic, and HBF lifetime.

    Source: §I · p. 2

  • Results

    HBF-based systems achieve 3.8–4.7× higher throughput than HBM-only systems, and LRU-10 extends HBF lifetime from about one year to over six years.

    Source: §I · p. 2, §V · p. 4

  • Takeaways & Limitations

    The results highlight the importance of write-aware KV-cache policy for sustainable HBF-based GR serving.

    Source: §V · p. 4

Abstract

from arXiv · show

Generative recommendation (GR) systems increasingly leverage user-level KV cache reuse to avoid recomputing long user histories. However, the growing KV cache capacity and bandwidth requirements introduce new challenges for memory system. High-Bandwidth Flash (HBF) provides a promising solution by offering substantially higher capacity than HBM while approaching HBM-class read bandwidth, enabling larger scale KV cache retention and improved serving throughput. Yet conventional Least-Recently-Used (LRU) KV cache management tightly couples KV cache writes with cache misses, generating excessive write traffic that rapidly exhausts flash endurance. In this work, we evaluate a write-aware KV cache policy based on admission-controlled LRU-K for HBF-based GR serving. By filtering low-reuse users before cache admission, LRU-K decouples KV cache writes from misses and significantly reduces unnecessary writes. We develop an analytical model to characterize GR serving performance, KV cache write traffic, and HBF lifetime, and evaluate performance across diverse memory systems and GR workloads. Our results show that HBF-based systems achieve 3.8 to 4.7 times higher throughput than HBM-only systems. Moreover, LRU-K extends HBF lifetime from about one year under conventional LRU to over six years with a moderate K=10, while maintaining comparable or even slightly improved throughput. These results highlight the importance of write aware KV cache policy for sustainable HBF-based GR serving.

I. INTRODUCTION

The section motivates HBF for GR’s growing KV-cache capacity and bandwidth needs, then evaluates admission-controlled LRU-K to reduce writes while preserving serving performance.

I. INTRODUCTION

  • For a 10K-user HSTU-10B workload, retaining the hottest 25% requires 3.62 TB, while meeting a 20 ms target for 8K-history requests requires 403 GB/s of KV-cache read bandwidth.The passage describes a mismatch: HBM has bandwidth but insufficient capacity at scale, while CPU memory has capacity but much lower bandwidth.Source: §I · p. 1
  • HBF offers over 10× HBM4’s per-stack capacity while approaching HBM-class read bandwidth, but NAND flash has limited write endurance.The passage gives 512 GB versus 36 GB per stack and 1.6 TB/s read bandwidth as examples.Source: §I · pp. 1–2
  • Conventional LRU admits even one-off users on misses and writes full caches, leaving a seven-stack HBF device at about one year under a 10K-user HSTU-10B workload.The passage identifies miss-driven cache churn as the source of accelerated flash wear.Source: §I · p. 2
  • The paper evaluates admission-controlled LRU-K, which admits user KV caches based on the K-th request and filters low-reuse users before admission.This decouples KV writes from cache misses and reduces unnecessary write traffic.Source: §I · p. 2
  • The authors develop an analytical model characterizing GR cache access, serving performance, write traffic, and HBF lifetime.Source: §I · p. 2
  • HBF-based systems achieve 3.8–4.7× higher throughput than HBM-only systems, while LRU-10 extends HBF lifetime from about one year to over six years.The reported lifetime extension maintains throughput.Source: §I · p. 2

II. GR INFERENCE WORKLOAD MODELING

GR inference processes a long user-interaction history and a shorter incremental sequence, reusing persisted user-level KV caches to avoid recomputing the history.

II. GR INFERENCE WORKLOAD MODELING

  • GR inference processes a user’s interaction sequence through prefill and candidate-item scoring through decode.Source: §II · p. 2
  • The interaction sequence combines a long historical sequence, such as thousands of tokens, with a much shorter sequence of newly arrived interactions.Source: §II · p. 2
  • Recent GR systems persist and reuse user-level KV caches across requests, using conventional LRU for cache management.This avoids recomputing the long historical prefix.Source: §II · p. 2

A. Serving Performance

The serving model distinguishes cache-hit and cache-miss paths and uses their latencies and hit rate to characterize average request latency and throughput.

A. Serving Performance

  • On a cache hit, the system reads historical KV and computes only incremental prefill and candidate decoding; on a miss, it recomputes historical prefill as well.The miss path also inserts the resulting full KV cache into memory.Source: A. Serving Performance · p. 2
  • The model computes operation times from forward-pass FLOPs and peak compute throughput, and data-transfer times from volume and bandwidth.It derives average request latency and QPS from these quantities.Source: A. Serving Performance · p. 2
  • A higher hit rate enabled by HBF’s larger capacity reduces average request latency and improves serving throughput by avoiding historical prefill.Source: A. Serving Performance · p. 2

B. KV Cache Write Traffic and HBF Lifetime

The write-traffic model links conventional LRU’s miss-driven full-cache writes to HBF lifetime, motivating admission control that decouples writes from misses.

B. KV Cache Write Traffic and HBF Lifetime

  • Under conventional LRU, misses write the full historical-plus-incremental KV cache, while hits write only incremental KV.Source: B. KV Cache Write Traffic and HBF… · p. 2
  • For tiered memory such as HBM+CPU, the model additionally captures tier-specific bandwidths, per-tier hit/miss behavior, and inter-tier data movement.The passage says those details are omitted for brevity.Source: B. KV Cache Write Traffic and HBF… · p. 2
  • The model defines the full and incremental KV sizes, usable HBF capacity, flash endurance, and workload request rate as inputs to write-volume and lifetime analysis.Usable KV capacity is HBF device capacity minus model weights.Source: B. KV Cache Write Traffic and HBF… · p. 3

III. WRITE-AWARE LRU-K CACHE POLICY

Admission-controlled LRU-K uses a user's request history to decide whether a missed KV cache is admitted, separating cache misses from writes. The model characterizes this policy through hit and admission probabilities tied to capacity, locality, and K.

III. WRITE-AWARE LRU-K CACHE POLICY

  • LRU-K admits a missed user's KV only when that user's K-th most recent request is newer than the oldest K-th request among cached entries.This criterion filters one-off or recently inactive users before their KV is persisted.Source: §III · p. 3

A. Controlled Admission and Write-Miss Decoupling

B. Impacts on HBF Lifetime and Serving Performance

LRU-K reduces KV write traffic by avoiding full-cache writes on miss-no-write requests, while increasing K slightly improves QPS by favoring active users with longer histories.

B. Impacts on HBF Lifetime and Serving Performance

A. Experimental Setup

The evaluation compares three memory configurations across HSTU and OpenOneRec workloads, with figures covering throughput under LRU-1 and HBF lifetime and write rate across LRU-K values.

A. Experimental Setup

  • The HBF-based configuration uses seven HBF stacks for KV cache and one HBM4 stack for frequently updated intermediate data.Each HBF stack is specified at 512 GB capacity, 1.6 TB/s read bandwidth, and 48 GB/s write bandwidth.Source: A. Experimental Setup · p. 3
  • Figure 5 compares throughput under LRU-1; Figure 6 compares HBF lifetime and write rate across LRU-K values.Source: Figure 5 · p. 4, Figure 6 · p. 4
  • Workloads include HSTU 1B, 10B, and 100B MoE, plus OpenOneRec 1.7B, 8B, and 100B MoE variants.Traffic spans diverse activity levels; histories are capped at 8K and candidate size is 500.Source: A. Experimental Setup · p. 4

B. Evaluation Results

HBF-based systems outperform HBM-only across models, while LRU-K substantially extends HBF lifetime. A moderate K also maintains or slightly improves throughput by favoring hotter users.

a) Throughput

  • HBF-based configurations achieve 3.8–4.7× higher QPS than HBM-only across all models.The comparison is under conventional LRU (LRU-1).Source: B. Evaluation Results · p. 4

b) HBF Lifetime

  • LRU-10 extends HBF lifetime from about one year with LRU-1 to over six years for all workloads.The lifetime gain is attributed mainly to reducing the full KV write probability.Source: B. Evaluation Results · p. 4
  • For HSTU-10B, increasing K from 1 to 10 reduces pwr from 0.101 to 0.014, lowering KV write traffic.LRU-K decouples cache insertions from misses, reducing the full-write probability.Source: B. Evaluation Results · p. 4
  • As K rises from 1 to 10, QPS slightly increases as LRU-K filters infrequent users and shifts cache residency toward hotter users.Retaining more active users with longer histories avoids costly prefill of long sequences; Figure 7 covers throughput and hit-rate distribution shifts for LRU-1 and LRU-10.Source: B. Evaluation Results · p. 4, Figure 7 · p. 4

V. CONCLUSION

The paper evaluates HBF for generative recommendation serving and addresses its write-endurance limitation with admission-controlled LRU-K. HBF improves throughput over HBM-only, while LRU-K extends HBF lifetime from about one year to over six years.

V. CONCLUSION

  • The work investigates HBF for GR serving, focusing on its write-endurance limitation.Source: §V · p. 4
  • Admission-controlled LRU-K selectively persists KV caches and decouples writes from cache misses.Source: §V · p. 4
  • HBF-based systems achieve 3.8–4.7× higher throughput than HBM-only, while LRU-K extends lifetime from about one year with LRU-1 to over six years with LRU-10.The paper concludes that write-aware KV cache policy is important for sustainable HBF-based GR serving.Source: §V · p. 4
Loading 2609.07175v1…