PaperlayerView exact revision on arXiv

Source-linked AI summary

FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration

Geraldo F. Oliveira, Arash Tavakkol, Xiangyu Zhu, Ahmet Caner Yüzügüler, Vamanan Arulchelvan, Lukas Cavigelli, Renzo Andri, Mohammad Sadrosadati, Jia Xinglei, Onur Mutlu, Zhou Ke, Shai Bergman, Ji Zhang

arXiv:2608.25062v1cs.ARcs.AIcs.DC

TL;DR

LLM inference is increasingly limited by accelerator memory capacity, particularly in small-node systems, while HBF offers multi-terabyte near-accelerator storage for model weights. FLINT is a workload-driven HBF substrate using dynamic burst buffering, phantom-plane refresh, and read-only translation; it improves throughput and energy efficiency across compared systems while reducing the GPU packages needed for a 50 ms TPOT SLO.

  • Problem

    Existing HBF proposals rely on static prefetching, expose refresh to the foreground read path, and retain unnecessary SSD-class write-management hardware for read-only weights.

  • Method

    FLINT co-designs a hardware burst-buffer controller, phantom-plane refresh, and a compact read-only FTL around the burst-granular, read-only behavior of LLM weight access.

  • Results

    FLINT improves decode throughput by 1,205×, 2.2×, and 6.2× versus SSD-equipped, HBM-only, and prior hybrid HBM+HBF GPU systems, respectively.

  • Takeaways & Limitations

    Workload-specialized HBF control turns on-package flash into a practical capacity tier for frontier LLM inference.

  • Takeaways & Limitations

    On-channel refresh can cost 54.2 ms for one block and stall foreground reads on the refreshed plane.

Abstract

from arXiv · show

LLM inference is increasingly constrained by accelerator memory capacity rather than compute throughput. This constraint is especially acute in single-accelerator and small-node inference systems, where limited on-package memory capacity restricts the size of deployable models. HBF is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (1) rely on coarse-grained static prefetching for LLM weights aiming to hide the microsecond-level read latency of the NAND flash device while maximizing HBF's read throughput, (2) expose NAND flash management tasks (e.g., refresh operations) to the accelerator-visible critical inference path, and (3) miss optimization opportunities to specialize and optimize the flash-management mechanisms to the workload behavior. Our goal is to design an efficient HBF substrate that integrates HBF as a memory-capacity tier alongside HBM while addressing these three challenges. To this end, we propose FLINT, a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (1) a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads aiming to utilize existing NAND flash buffers while sustaining high HBF bandwidth, (2) a phantom-plane refresh mechanism, which removes refresh from the critical inference path by moving refresh-related NAND flash operations outside the read foreground back via low-cost resource duplication, and (3) a read-only FTL, which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.

1. Introduction

LLM weight growth is creating a memory-capacity bottleneck, especially for small-node inference, motivating HBF as a near-accelerator capacity tier. FLINT addresses prior HBF limitations with workload-driven buffering, refresh, and address translation mechanisms.

  • Motivation: Frontier LLM parameter growth makes inference a memory-capacity problem because weights must remain resident near the accelerator.Recent high-end GPUs provide only 80–192 GB of on-package memory, while frontier models reach hundreds of billions to trillions of parameters.
  • Motivation: Adding accelerators solely for model capacity is often impractical in single-accelerator and small-node settings, despite sufficient compute and memory bandwidth.Partitioning models across accelerators also increases deployment cost and can introduce interconnect-induced bottlenecks.
  • HBF Opportunity: HBF provides multi-terabyte near-accelerator capacity with TB/s-class bandwidth by stacking dense 3D NAND flash dies in an HBM-like form factor.D2D links can daisy-chain HBF with HBM while expanding on-package capacity.
  • Prior Limitations: Prior HBF designs use dedicated SRAM buffers and static coarse-grained prefetching that cannot reliably align burst issuance with runtime cache-line demand.Runtime MoE routing, token-dependent reuse, and accelerator request interleaving make burst consumption difficult to predict statically; existing designs also leave refresh management unresolved.
  • FLINT: FLINT dynamically orders and coalesces fine-grained reads into coordinated bursts driven by the accelerator’s runtime access stream.Its burst-buffer controller uses existing HBF page and cache buffers, while phantom-plane refresh and a read-only FTL specialize maintenance and translation for read-only weights.
  • Evaluation: FLINT improves decode throughput by 1,205×, 2.2×, and 6.2× versus SSD-equipped, HBM-only, and prior hybrid HBM+HBF GPU systems, respectively.It reduces energy consumption by 408×, 1.1×, and 6.8× against those baselines, meets a 50 ms TPOT SLO with 3.1× fewer GPU packages, and incurs 3.1% HBF-die area cost.

2. Background

HBF integrates NAND flash into the accelerator package as a high-capacity memory tier alongside HBM. Its organization and operation explain both its bandwidth potential and the latency and maintenance challenges that system designs must manage.

  • HBF-Enabled System: An HBF-enabled package co-packages xPUs, HBM stacks, and HBF stacks on a silicon interposer.An example HBF stack provides 512–1024 GB capacity and 1.6–2 TB/s aggregate bandwidth.
  • HBF-Enabled System: HBF expands package capacity while exposing HBM and HBF through a unified physical address space.The HBM base die directs latency-sensitive or write-mostly state to HBM and read-only inference state, such as weights, to HBF.
  • HBF Organization: An HBF stack contains multiple flash dies, each die contains many planes, and each plane contains NAND blocks plus page and cache buffers.The page buffer holds the page being sensed or programmed, while the cache buffer holds the previous sensed page.
  • NAND Flash Operation: NAND flash provides page read, page program, and block erase as its primitive operations.SLC page reads take 1–2 µs, page programs approximately 50 µs, and block erases approximately 3 ms.
  • Flash Reliability: Repeated reads and elapsed time shift flash-cell threshold voltages through read disturbance and retention loss, eventually causing read errors.Flash controllers periodically refresh blocks to bound the resulting bit-error rate.

3. Motivation

Capacity-driven LLM inference incurs substantial multi-GPU communication, synchronization, and memory-system costs, while existing HBF designs leave bandwidth, refresh, and flash-management inefficiencies unresolved.

  • Capacity-Forced Scaled-Out Inference: Inter-package communication and synchronization consume 52%–85% of per-token time at batch sizes 1–4, remaining 22%–61% at batch size 64.Layer-barrier waiting accounts for 51%–79% of per-token time at batch sizes 1–4, while accelerator compute occupies only 0.1%–1.9%.
  • Capacity-Forced Scaled-Out Inference: Expert-routing imbalance makes the busiest GPU package read 1.4×–4× the routed-expert weight bytes of the mean package at batch sizes 1–4.The imbalance falls to 1.1× at batch size 64 as larger batches distribute routed tokens more evenly.
  • Static Prefetch Underutilizes the HBF Channel: Static HBF prefetching wastes 86%–96% of traffic on MoE models across batch sizes, and perfect prefetch coverage changes throughput by at most 3% at batch sizes 16–64.At batch size 1, perfect coverage recovers 2.0×–2.8× on three MoE models but reduces DeepSeek-V3 throughput to 0.83×–0.92×.
  • Static Prefetch Underutilizes the HBF Channel: Static prefetches fail because some bursts return before a layer-ahead hint can act, while larger active sets cause staging-buffer thrashing.At batch size 1, median re-fetch return times are 0.12×–0.45× of a layer’s execution time; at batch size 64, buffer thrashing dominates.
  • On-Channel Refresh Saturates the HBF Channel: A single NAND block refresh takes 54.2 ms, and aligned burst refresh can stall the HBF channel for 26 s to 22 min.Distributed refresh reduces burst-refresh stalls by spreading work over time, but its steady refresh stream still competes with foreground reads.
  • SSD-Class FTL Overprovisions the Read Path: SSD-class FTLs retain write-oriented hardware for updates, garbage collection, wear leveling, allocation, and crash consistency despite read-only inference weights.The paper identifies these mechanisms as unnecessary when weights are programmed once at deployment and remain read-only during inference.

4. FLINT Overview

FLINT drives HBF from the accelerator’s dynamic read stream using a burst-buffer controller, phantom-plane refresh, and a compact read-only translation table. These mechanisms coalesce and pipeline weight reads, move refresh off the foreground path, and map logical bursts to physical locations.

  • FLINT Overview: FLINT uses three mechanisms: a hardware burst-buffer controller, phantom-plane refresh, and a burst-granular read-only FTL.The design exploits burst-granular, read-only LLM weight access rather than static compile-time predictions.
  • Burst-Buffer Controller: The burst-buffer controller groups fine-grained cache-line requests by physical page and block, then schedules them as plane-parallel HBF bursts.Its cache request queue, mapping table, and scheduler bridge accelerator cache-line requests to NAND page reads.
  • Burst-Buffer Controller: Distinct page and cache buffers overlap the next burst’s NAND sensing with the current burst’s drain, sustaining one complete burst per drain interval.This pipeline avoids adding a dedicated HBM-side SRAM staging buffer.
  • Phantom-Plane Refresh: Phantom-plane refresh adds one physical plane per die, allowing N live planes to serve reads while the extra plane receives ECC-corrected block copies in the background.The phantom role rotates across planes, with block-granular remapping as copies complete.
  • Phantom-Plane Refresh: The refresh design sets a rotation step of at least 12.5 s and completes a full pass across all planes in about seven minutes.The measured hot-block demand of 0.86 GB/s per stack fits within the 1.28 GB/s program bandwidth of sixteen phantom planes.
  • Read-Only FTL: FLINT’s read-only FTL stores one physical (block, page) coordinate per logical 2 MB burst, requiring about 1 MB of SRAM for a 512 GB stack.Fixed burst placement makes slice-to-plane assignment implicit and removes arbitrary-write support.
  • Power-Loss Recovery: FLINT treats every power-up as a clean installation rather than preserving resident weights and persistent refresh state after an unplanned outage.This assumption avoids reintroducing write-path support for unbounded outage durations and retention drift.

5. Methodology

FLINT is evaluated with a trace-driven simulator modeling the accelerator’s memory hierarchy, HBF behavior, package communication, and synchronization. Experiments cover six production LLMs across dense and MoE workloads, multiple batch sizes and context lengths, and three deployment baselines.

  • Simulation Framework: The simulator models five weight-read tiers: SMs, on-die L2, HBM, staging buffers, and HBF.It also charges inter-package transfers and synchronization stalls in multi-package configurations.
  • Simulation Framework: The HBM model is calibrated against Ramulator 2.0, which reports sustained-to-peak bandwidth of 67.8–69.9 % across models, batch sizes, and context lengths.Arithmetic is derived from trace tensor shapes and executed using a machine-utilization factor on a B200-class accelerator.
  • Workloads: The workload suite includes six production LLMs: five MoE models and the dense Llama 3.1-405B.Traces use native precision, batch sizes {1, 4, 16, 64}, and context lengths {2, 8, 32, 128} K tokens.
  • Baselines: The baselines are HBM+SSD, HBM-only, and H3, a one-package HBF design using two LHB slots, FIFO eviction, and compiler-emitted layer-ahead hints.HBM-only uses the fewest packages that hold the weights alone and marks KV-cache-infeasible configurations.

6. Evaluation

Across dense and MoE LLM inference, FLINT improves throughput and energy relative to SSD, HBM-only, and prior hybrid systems while reducing the GPU packages needed to meet the SLO. Its burst buffering delivers useful HBF bandwidth, phantom-plane refresh removes refresh overhead, and the estimated area cost is small.

  • HBF Bandwidth: 90–97 % of fetched HBF traffic is consumed, yielding 1.9–3.6 TB/s useful MoE bandwidth, 4.0×–14.3× H3’s useful bandwidth.Across all six models, the useful-bandwidth improvement is 6.2× on average.
  • Performance Analysis: 2.2× higher per-GPU decode throughput than HBM-only is achieved across all six models, with a measured range of 1.5×–3.1×.The HBM-only baseline spends 74–85 % of decode time waiting at layer barriers.
  • Energy Analysis: 0.72×–0.90× the HBM-only energy per token is consumed on average across five MoE models.At batch size 1, FLINT uses 0.45×–0.73× the HBM-only energy per token.
  • Capacity Scaling: A single FLINT GPU package serves each of five MoE models within the 50 ms TPOT SLO at batch size 1, while HBM min-fit requires four to eight packages for weights.At batch size 4, FLINT still serves five of six models with one or two packages.
  • Refresh Execution Cost: FLINT’s off-path refresh matches refresh-free decode throughput, whereas in-place refresh costs 20× on average across models.In-place burst refresh causes a 3,700× average stall at the token where refreshes land, versus 4.8× averaged over the refresh period.
  • Area Overhead: FLINT’s area overhead is estimated as 3.1 % on an HBF die and 3.9 mm2 on the HBF base die at 7 nm.The estimates include added die storage and base-die control and storage structures.

7. Related Work

FLINT differs from prior HBF and non-volatile-memory inference systems by using runtime-driven burst buffering, on-package HBF, GPU-resident computation, and a compact read-only translation table. This specialization targets immutable, burst-granular LLM weight access while removing storage-class write-path machinery from foreground inference.

  • HBF-Based Systems: FLINT is presented as the first HBF substrate sustaining high HBF bandwidth without a dedicated HBM-side SRAM staging buffer or compiler-emitted prefetch hints.Its design co-optimizes runtime-aware burst buffering, phantom-plane refresh, and read-only FTL.
  • HBF-Based Systems: Prior HBF systems commonly use HBM-side SRAM staging buffers and compiler- or programmer-emitted layer-boundary hints.FLINT instead drives HBF from the dynamic accelerator read stream.
  • Non-Volatile Memory for LLM Inference: Unlike several non-volatile-memory systems, FLINT places the capacity tier on-package in HBF, keeps computation on the GPU, and reaches HBF through the HBM-side D2D link.This distinguishes FLINT from off-package storage and compute-in-storage designs.
  • Non-Volatile Memory for LLM Inference: FLINT replaces storage-class FTL support with burst-granular read-only translation, eliminating OS-mediated page faults, page-granular metadata, and write-path machinery from foreground inference.The table is specialized for immutable LLM weights.

8. Conclusion

FLINT specializes HBF control for the burst-granular, read-only behavior of LLM inference weights, turning on-package flash into a practical capacity tier for frontier LLM inference.

  • FLINT uses a hardware burst-buffer controller, phantom-plane refresh, and a read-only FTL to specialize HBF for LLM inference weights.The controller stages bursts in existing HBF die buffers, phantom-plane refresh keeps NAND maintenance off foreground reads, and the FTL uses compact burst-granular address translation.
  • Workload-specialized HBF control turns on-package flash into a practical capacity tier for frontier LLM inference.
Loading 2608.25062v1…