PaperlayerView exact revision on arXiv

Source-linked AI summary

HAWK: Heterogeneity-Aware KV Cache Transfer for Efficient GPU-NPU Phase-Disaggregated LLM Inference

Junhyeok Park, Jungeun Yoon, Kyungmin Kim, Hunsoon Lee, Shinyoung Ahn, Kangho Kim

arXiv:2610.29873v1cs.ARcs.DC

TL;DR

GPU prefill and NPU decode offer complementary efficiency, but heterogeneous serving must transfer KV caches across devices with different communication mechanisms and tensor layouts. HAWK provides a direct RDMA path and layout-aware transfer, and its heterogeneous system remains within 7.88% of GPU-only performance while achieving higher power and cost efficiency.

  • Problem

    GPU and NPU strengths differ across prefill and decode, while transferring KV caches between them is complicated by distinct communication mechanisms and tensor layouts.

    Source: §1 · p. 1

  • Method

    HAWK establishes a common RDMA path between GPU and NPU memory and uses transfer-aware transformation and NPU-side layout-aware buffers to produce tensors directly usable for decode.

    Source: §1 · p. 2

  • Results

    Within 7.88% performance difference from a GPU-only system, the heterogeneous system achieves up to 1.26× higher power efficiency and 1.72× higher cost efficiency under the same SLO constraints.

    Source: §7 · p. 11

  • Takeaways & Limitations

    Transferred KV tensors can be consumed directly for decode without post-transfer conversion.

    Source: §1 · p. 2

    The conventional single-contiguous-buffer NPU configuration is excluded from quantitative comparisons after serving failed with out-of-memory errors from DRAM-channel memory fragmentation.

    Source: §5.1.3 · p. 8

Abstract

from arXiv · show

Large language model (LLM) inference exhibits distinct computational characteristics across its two phases: prefill is compute-intensive, whereas decode is more memory-bound. As a result, AI accelerators achieve varying levels of efficiency across the two phases. GPUs are well suited to prefill, but are often underutilized during decode. By contrast, NPUs can utilize their compute resources more effectively during decode. Motivated by this observation, we explore a heterogeneous architecture that integrates GPUs and NPUs, mapping prefill to GPUs and decode to NPUs to achieve high performance and efficiency. However, realizing such heterogeneous integration is challenging because LLM phase disaggregation requires KV cache transfer from prefill to decode (that is, from GPUs to NPUs), which is complicated by differences in inter-device communication mechanisms and KV cache tensor layouts. To address this, we propose HAWK, a transfer mechanism that enables rapid KV cache transfer across heterogeneous devices. HAWK establishes a remote direct memory access (RDMA) path for direct cross-device transfers while handling inter-device tensor layout discrepancies as part of the transfer process. In particular, HAWK allocates tensor-layout-aware RDMA buffers on the NPU side to match the target tensor layout. Leveraging these buffers, HAWK orchestrates data transfer from a conventional contiguous GPU RDMA buffer to the corresponding NPU buffers, allowing transferred tensors to be consumed directly without additional data rearrangement while maintaining transfer efficiency. We implement and evaluate HAWK on a system built with commercially available GPUs and NPUs, demonstrating performance within 7.88% of a GPU-only system under the same SLO constraints while achieving up to 1.26$\times$ higher power efficiency and 1.72$\times$ higher cost efficiency.

1 Introduction

GPU–NPU phase disaggregation pairs GPUs for prefill with NPUs for decode, but heterogeneous transfers must address communication and tensor-layout differences. HAWK provides a direct RDMA-based transfer mechanism, and the resulting system approaches GPU-only performance while improving power and cost efficiency.

1 Introduction

  • Heterogeneous transfer must address both device-specific communication mechanisms and tensor-layout discrepancies that can require additional data rearrangement.These differences complicate interoperability and can increase transfer latency.Source: §1 · p. 2
  • Within 7.88% performance difference from a GPU-only system, HAWK achieves up to 1.26× higher power efficiency and 1.72× higher cost efficiency under the same SLO constraints.These are system-level results for the heterogeneous GPU–NPU setup.Source: §1 · p. 2
  • GPUs achieve high compute utilization during prefill but are underutilized during decode, whereas NPUs offer higher decode compute utilization with comparable decode performance.The paper characterizes this phase-dependent behavior as motivation for heterogeneous serving.Source: §1 · p. 2
  • The proposed heterogeneous system assigns prefill to GPUs and decode to NPUs, while identifying cross-device transfer as a critical bottleneck.The design aims to combine high performance and efficiency across the two phases.Source: §1 · p. 2
  • HAWK uses an RDMA-based path and transfer-aware tensor transformation with tensor-layout-aware buffers to handle cross-device layout mismatches.The NPU-side buffers match the target layout, allowing transferred tensors to be consumed without additional rearrangement.Source: §1 · p. 2

2 Background

The background describes the decoder-only Transformer’s prefill and autoregressive decode flow, including how attention activations are reused through a growing KV cache. It then relates phase-specific input shapes to GEMM-like prefill and GEMV-like decode computation.

2.1 LLM Inference

  • Prefill embeds the input tokens, processes them through decoder layers and the LM head, while decode feeds each selected token back until EOS or the maximum generation length.The two phases share the same layer sequence but differ in the number of tokens processed.Source: §2.1 · p. 2, Figure 1 · p. 3
  • Key and Value activations remain unchanged across later decoding steps, so the KV cache reuses them and grows as new token activations are appended.This avoids recomputing those activations during subsequent decoding.Source: §2.1 · pp. 2–3

2.2 Characteristics of Prefill and Decode

  • Input length shapes the dominant computation: prefill processes all input tokens as GEMM, while one-token decode maps to GEMV with lower arithmetic intensity and data reuse.Decode also fetches KV caches from previously processed tokens, adding memory traffic.Source: Table 2 · p. 3, §2.2 · p. 3

3 Motivation

GPU efficiency is high during prefill but falls during decode, while NPUs show stronger decode utilization and power efficiency. This motivates assigning prefill to GPUs and decode to NPUs, but heterogeneous serving must address cross-device transfer and tensor-layout differences.

3.1 Phase-Dependent GPU Efficiency

  • During decode, GPU tensor-core utilization is ∼35.90% versus ∼92.92% during prefill, while DRAM utilization is ∼64.09% versus ∼19.26%.The QKV projection kernel is representative; other decoder-layer kernels also show low computational efficiency during decode.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Potential of NPUs for Efficient Decode

  • During decode, NPUs exceed GPU compute utilization across evaluated kernels and achieve higher power efficiency for attention and FFN projections.The evaluated NPU uses fewer, DRAM-proximate compute cores to accommodate limited parallelism and GEMV-like workloads; only one NPU platform was evaluated.Source: §3.2 · p. 4, Table 4 · p. 4

3.3 LLM Disaggregation with GPUs and NPUs

  • Phase disaggregation sends requests to a prefill instance, which generates the first token and KV cache before transferring them to a decode instance.The paper maps prefill to GPUs and decode to NPUs based on their phase-specific characteristics.Source: §3.3 · p. 4

3.4 Challenges: Handling Heterogeneity

  • GPU–NPU serving must reconcile distinct communication mechanisms and tensor layouts; host staging adds 1.83× to 5.25× the latency of direct transfer as transfer size grows from 10 MB to 1.2 GB.Forcing a shared layout can reduce device-specific memory efficiency and require compute-kernel changes.Source: §3.4 · pp. 4–5

4 Heterogeneity-Aware KV Cache Transfer

HAWK supports heterogeneous GPU–NPU inference by providing a direct transfer path for KV caches and handling layout differences during transfer. Its NPU-side RDMA buffers match the target tensor layout so received data can be used directly during decode.

4 Heterogeneity-Aware KV Cache Transfer

  • HAWK establishes a common RDMA path between GPU and NPU memory and transforms GPU KV tensors into transfer-friendly aligned tensors for the target NPU layout.It transfers the data into multiple layout-aware NPU RDMA buffers, allowing the received tensors to be consumed during decode without additional layout transformation.Source: §4 · p. 5, Figure 5 · p. 5
  • Device-specific peer memory support lets the NIC access registered GPU and NPU memory directly, without host intervention.HAWK extends the NPU driver with a peer memory client and uses the Linux RDMA stack.Source: Figure 6 · p. 5, §4 · p. 5

4.1 Building a Direct Data Path Across Devices

HAWK connects GPU and NPU device memory through a common Linux RDMA stack, using device-specific peer memory clients to expose memory mappings for registration and direct NIC access.

4.1 Building a Direct Data Path Across Devices

  • GPU and NPU peer memory clients expose device-memory mappings to a common Linux RDMA stack, enabling NIC access to registered memory and runtime transfer engines to orchestrate data movement.The GPU side uses a peer memory client, while HAWK adds a corresponding client to the NPU driver.Source: §4.1 · p. 5

4.2 Tensor-Layout-Aware Buffer Allocation

NPU tensors use channel-specific memory layouts that differ from a conventional contiguous RDMA buffer, creating transfer and allocation problems. HAWK allocates RDMA buffers per channel to match the target tensor layout, with a bounded increase in buffer-management and request overhead.

4.2 Tensor-Layout-Aware Buffer Allocation

  • Conventional NPU RDMA buffers can misalign data with tensor layouts and imbalance channel capacity, potentially blocking later tensor allocations or reducing data-movement efficiency.NPU tensors distribute chunks across channels using explicit placement, unlike a virtually contiguous buffer.Source: §4.2 · p. 6
  • HAWK allocates one virtually contiguous RDMA buffer per NPU DRAM channel, matching the target tensor layout and enabling direct consumption or limited intra-channel movement.This avoids costly rearrangement across multiple DRAM channels.Source: §4.2 · p. 6
  • Per-channel buffers increase buffer-management and RDMA-request overhead, but their count is bounded by the NPU's typically ten-or-fewer DRAM channels.Each per-channel buffer remains contiguous, keeping the added overhead modest relative to a single-buffer design.Source: §4.2 · p. 6

4.3 Transfer-Aware Tensor Transformation

HAWK divides GPU-to-NPU tensor-layout alignment across two steps, arranging data for larger RDMA transfers and completing the target layout during transfer. This avoids many small requests and post-transfer rearrangement.

4.3 Transfer-Aware Tensor Transformation

  • HAWK first reorders GPU tensor dimensions and elements, grouping portions for each per-channel NPU buffer contiguously while partially aligning the target layout.The intermediate GPU tensor is arranged into large contiguous chunks for each NPU RDMA buffer; remaining alignment occurs during transfer.Source: §4.3 · p. 6, Figure 8 · p. 7
  • This arrangement increases individual RDMA transfer sizes and reduces request count; prefill has bandwidth headroom for the added GPU transformation work.HAWK performs this transformation on the GPU prefill instance before transfer, where device-memory bandwidth is less pressured than during decode.Source: §4.3 · p. 6
  • HAWK pairs each contiguous GPU portion with its per-channel NPU buffer, completing layout alignment during transfer so the tensor needs no post-transfer transformation.The alternative of mapping small portions individually would create many few-hundred-byte requests and greater protocol and request-processing overhead.Source: §4.3 · p. 7

4.4 Putting It All Together

HAWK transfers a requested KV cache from the GPU prefill instance to the NPU decode instance using metadata to pair GPU buffer regions with per-channel NPU buffers. Batched RDMA requests are followed by tensor reconstruction and buffer cleanup.

4.4 Putting It All Together

  • The decode instance requests a KV cache by request ID, after which prefill retrieves and transforms it, reserves GPU RDMA-buffer space, copies the tensor, and sends buffer metadata.The metadata includes the total size and buffer offset.Source: §4.4 · p. 7
  • Decode uses the metadata to pair GPU-buffer regions with per-channel NPU buffers and submits the RDMA requests as a batch rather than sequentially.After transfer completes, decode reconstructs the NPU KV-cache tensor from the received buffers.Source: §4.4 · p. 7
  • After transfer completion, both instances release their reserved RDMA-buffer regions.Decode notifies prefill before the respective receive and send buffers are released.Source: §4.4 · p. 7

4.5 Discussion

HAWK’s tensor-transfer design is adaptable to different NPU layouts, other disaggregation schemes that transfer tensors, and transfers from NPUs back to GPUs.

4.5.1 Extensibility Across NPU Architectures.

  • HAWK’s prototype targets an optimized tiled layout, but its core mechanisms can be adapted to other tiled or blocked NPU layouts through device-specific tensor-aware RDMA-buffer allocation.Tensor layouts can differ across devices even when the framework-level tensor is the same.Source: §4.5.1 · p. 7

4.5.2 Adaptability to Other Forms of Disaggregation.

  • HAWK can transfer intermediate activations in operator-level disaggregation, where GPU attention is followed by NPU operators.Activations are typically smaller than KV caches, making fixed transformation costs more significant, and dependent operators limit latency-hiding overlap.Source: §4.5.2 · pp. 7–8

4.5.3 Reverse Transfer.

  • For NPU-to-GPU transfer, the transformation can run on the NPU, with per-channel NPU buffers mapped to a virtually contiguous GPU buffer to complete layout alignment during transfer.This preserves fewer, larger RDMA transfers and allows direct GPU use without post-transfer layout transformation.Source: §4.5.3 · p. 8

5 Evaluation

Across the evaluated workloads, GN-HAWK approaches GPU-only serving performance at 90% SLO attainment while improving power and cost efficiency. The evaluation covers KV-cache transfer, online serving, and an H100 comparison.

5.1 Evaluation Setup

  • Evaluation uses A100 prefill and p150a decode on separate 200-Gbps nodes, vLLM with Llama 3.1 70B, three baselines plus GN-HAWK, and synthetic and serving workloads; stack coupling limits model coverage.The direct single-buffer NPU RDMA configuration was excluded from quantitative comparison after an out-of-memory error during serving: imbalanced DRAM-channel use left insufficient capacity for subsequent tensor allocations.Source: §5.1.1 · p. 8, §5.1.2 · p. 8, §5.1.3 · p. 8

5.2 End-to-End KV Cache Transfer Latency

  • HAWK transfer latency is within 1% of GG-Direct, while GN-Host latency falls from 8.91× to 3.79× that of GG-Host as transfer size grows from 10 MB to 1280 MB.The GN-HAWK latency gap relative to GN-Host becomes negligible at larger transfer sizes, which are typical in practical serving workloads.Source: Figure 9 · p. 9, §5.2 · p. 9

5.3.1 TTFT.

  • At 90% SLO attainment, GN-HAWK sustains request rates within 2.90% and 5.90% of GG-Direct for Chatbot and Summarization, respectively.At higher Summarization request rates, the TTFT gap is attributed to request queuing associated with NPU software-stack integration.Source: Figure 10 · p. 9, §5.3.1 · p. 9

5.3.2 TPOT.

  • For Summarization, GN-HAWK reaches the TPOT SLO threshold at 0.60 requests/s; the TPOT gap narrows substantially at 0.80 requests/s.For Chatbot, both GN-HAWK and GG-Direct remain below the TPOT SLO threshold.Source: Figure 11 · p. 9, §5.3.2 · p. 9

5.3.3 SLO Attainment.

  • For Summarization, GN-HAWK reaches 90% SLO attainment at 0.60 requests/s versus GG-Direct’s 0.65, a 7.69% request-rate difference.For Chatbot, both configurations maintain 90% SLO attainment up to 0.65 requests/s.Source: Figure 12 · p. 10, §5.3.3 · p. 10

5.3.4 Output Throughput.

  • GN-HAWK achieves 0.48% and 7.88% lower decode throughput than GG-Direct for Chatbot and Summarization, respectively, at 90% SLO attainment.The remaining gap mainly stems from decode-device performance differences; GN-Host is omitted because it fails to attain 90% SLO even at the lowest evaluated request rate.Source: Figure 13 · p. 10, §5.3.4 · p. 10

5.4 Serving Efficiency

  • GN-HAWK achieves 1.26× and 1.15× higher power efficiency than GG-Direct for Chatbot and Summarization, and 1.72× and 1.61× higher cost efficiency, respectively.The cost-efficiency comparison is reported as a reference because device prices vary with market conditions.Source: §5.4.1 · p. 10, Table 8 · p. 10, §5.4.2 · p. 11

5.4.3 Comparison with Another GPU Architecture.

  • Against GGH-Direct, GN-HAWK achieves 1.03× higher power efficiency and up to 2.76× higher cost efficiency, with throughput within 6.53%.The comparison uses H100 GPUs for GGH-Direct decoding; efficiency gains depend on the GPU and NPU platforms used.Source: Table 9 · p. 11, §5.4.3 · p. 11

6 Related Work

Prior work optimizes phase-disaggregated serving largely within GPU-based systems and explores heterogeneous inference across several accelerator pairings. This work extends the focus to coordinating GPUs and NPUs across multiple servers.

6.1 Phase-Disaggregation

  • Prior phase-disaggregated serving work explores resource provisioning, placement, cluster scheduling, prefill-decode ratio adaptation, and KV-cache management and transfer.These efforts primarily optimize disaggregated serving within GPU-based systems.Source: §6.1 · p. 11

6.2 Other Forms of Heterogeneous Integration

  • Earlier heterogeneous-inference studies combine GPUs with CPUs, NPUs, or PIM/PNM accelerators, and also examine integrated NPU-PIM systems.Their approaches include workload partitioning, offloading, scheduling, and disaggregation to improve inference performance and efficiency.Source: §6.2 · p. 11
  • HAWK extends prior work by addressing the practical challenges of coordinating GPUs and NPUs across multiple servers.The paper develops system-level solutions for this setting.Source: §6.2 · p. 11, §6.1 · p. 11

7 Conclusion

HAWK enables GPU-prefill and NPU-decode inference by transferring KV caches across the heterogeneous devices. Under the same SLO constraints, it maintains performance within 7.88% of a GPU-only system while improving power and cost efficiency.

7 Conclusion

  • Within 7.88% of a GPU-only system’s performance under the same SLO constraints, HAWK achieves up to 1.26× higher power efficiency and 1.72× higher cost efficiency.Source: §7 · p. 11
  • HAWK transfers data directly over a common RDMA stack and addresses tensor-layout differences with layout-aware buffer allocation and transfer-aware tensor transformation.Source: §7 · p. 11
  • The proposed system assigns prefill to GPUs and decode to NPUs to pursue high performance and efficiency.Source: §7 · p. 11

Related papers

Loading 2610.29873v1…