Source-linked AI summary
PATTON: Enabling Commodity PIM for Production LLM Serving
Hangyeol Kim, Sanghyun Lee, Teokkyu Suh, Joo-Young Kim
TL;DR
Production LLM serving dynamically manages KV cache blocks, but commodity PIM needs explicit block placements and commands, and Value layouts trade off GEMV efficiency, write efficiency, and capacity. PATTON combines hierarchical granule allocation with a Commit Zone to coordinate those operations, achieving 1.95× speedup and 4.83× higher energy efficiency across attention and runtime-induced prefill recomputation over evaluated baselines.
Problem
Production serving engines dynamically manage logical KV blocks, but commodity PIM requires efficient physical allocation, block-to-address mapping, and command generation; Value placement also trades off GEMV, write, and capacity efficiency.
Source: §1 · pp. 1–2Method
PATTON maps block-sized Key and Value granules to logical blocks, groups them into coarser granules, and stages partial Value blocks in a Commit Zone before committing them to GEMV-optimized locations.
Source: §1 · pp. 2–3Results
1.95× speedup and 4.83× higher energy efficiency over evaluated baselines across attention execution and runtime-induced prefill recomputation, without modifying the underlying PIM architecture.
Source: §1 · p. 3Takeaways & Limitations
PATTON supports dynamic KV allocation, prefix sharing and caching, reclamation, and attention execution on commodity PIM without modifying its processing units.
Source: §6 · p. 12Prior approaches leave exact block and token layouts or repeated multi-request allocation and reclamation unspecified, while some write-efficient layouts require processing-unit functionality beyond commodity MAC.
Source: §3.3 · pp. 6–7
Abstract
from arXiv · showhide
Processing-in-Memory (PIM) is promising for accelerating memory-bound decode attention, but attention acceleration alone is insufficient for production LLM serving, where engines dynamically allocate, populate, share, cache, and reclaim logical KV cache blocks. Supporting this lifecycle on commodity PIM requires efficient physical memory allocation, block-to-address mapping, and command generation. For the Value cache, these requirements create a fundamental conflict among GEMV efficiency, single-token write efficiency, and memory capacity: GEMV-optimized layouts scatter newly generated Value vectors across rows, making writes costly, while finer-grained memory sharing improves capacity utilization but fragments GEMV reductions. We present PATTON, a PIM runtime that integrates production LLM serving engines with commodity PIM. PATTON introduces hierarchical granule allocation: block-sized Key and Value granules map one-to-one to logical token blocks, fixing their physical placements and commands, while coarser granules group blocks for efficient GEMV execution and memory utilization. A Commit Zone stages partial Value blocks for efficient single-token writes before committing them to GEMV-optimized locations. PATTON tracks these placements to generate KV cache writes and QK-transpose/SV commands. Across attention execution and runtime-induced prefill recomputation, PATTON achieves an average 1.95x speedup and 4.83x higher energy efficiency over evaluated baselines, requires no PIM processing-unit modifications, and maintains a KV cache hit rate comparable to the native GPU KV cache in vLLM.
1 Introduction
Production LLM serving dynamically manages KV cache blocks, so commodity PIM must support that lifecycle as well as accelerate attention. PATTON coordinates block placement, memory allocation, and command generation to address the Value-cache trade-offs among GEMV, writes, and capacity.
1 Introduction
- Production serving engines dynamically allocate, populate, share, cache, and reclaim KV blocks, requiring hardware support beyond attention computation.The engines use block-based cache management, including prefix sharing and caching.Source: §1 · p. 1
- The runtime must translate logical block IDs and token offsets into physical PIM addresses and operation-specific commands, a capability existing approaches do not fully provide.The mapping also determines Key and Value write patterns, attention MAC scheduling, and memory utilization.Source: §1 · pp. 1–2
- Value layouts that keep token reductions contiguous for GEMV scatter new-token vectors across rows, while row sharing can fragment reductions and reserving rows wastes capacity.These competing layouts affect write latency, GEMV efficiency, and retention of cached prefixes.Source: §1 · p. 2
- PATTON maps block-sized Key and Value granules one-to-one to logical blocks, groups them into coarser GEMV- and capacity-oriented granules, and stages partial Value blocks in a Commit Zone before committing them.The runtime records placements to generate KV writes and QK-transpose/SV commands.Source: §1 · pp. 2–3
- 1.95× speedup and 4.83× higher energy efficiency over evaluated baselines across attention execution and runtime-induced prefill recomputation, without modifying the PIM architecture.PATTON is implemented in vLLM and targets AiM.Source: §1 · p. 3
2 Background
Commodity PIM executes GEMV through parallel bank-level MAC units, while serving engines manage KV caches through demand-allocated logical blocks. Prefix sharing and caching make earlier context reusable, especially across conversation turns.
2.1 GEMV Execution on Commodity PIM
- Banks execute output elements in parallel while consecutive columns within each bank row hold the GEMV reduction dimension.AiM supplies a shared input operand from the Global Buffer to bank MAC units; its commands load inputs, invoke MACs, and retrieve results.Source: §2.1 · p. 3
2.2 AiM Command Set
- AiM commands load operands and initialize registers, invoke bank MAC units, and read results; minimizing repeated transfers and retrievals reduces PIM I/O overhead.RDCP and WRCP support internal data movement without changing the MAC datapath.Source: §2.2 · p. 3
2.3 PagedAttention and Prefix Sharing
- PagedAttention allocates fixed-size physical KV blocks on demand and maps logical blocks through a per-request table, while shared full prefix blocks remain read-only.This avoids reserving maximum context length, leaving unused space mainly in each request’s final partial block.Source: §2.3 · p. 3
2.4 Multi-Turn LLM Serving
- In multi-turn serving, prior conversation context becomes the next request’s prefix; without prefix caching, cache misses account for about 90% of prefill work.Prefix caching retains completed requests’ blocks for later reuse, while an evicted needed prefix must be recomputed.Source: §2.4 · pp. 3–4, Figure 3 · p. 4
3 Motivation and Observations
Production PIM serving must translate engines’ logical KV-block operations into physical placements and commands while preserving capacity and attention efficiency. PATTON’s requirements arise from different Key and Value layouts, the Value cache’s write–GEMV–capacity conflict, and gaps in prior systems.
3.1 Runtime Responsibilities in Production Serving
- vLLM supplies logical block IDs and token offsets; the runtime records physical placements to generate Key and Value writes, form QKᵀ and SV operands, and issue MAC commands.The metadata also supports cache-hit checks and accumulator retrieval and reinitialization at operation boundaries.Source: §3.1 · p. 4
3.2 Key and Value Placement Tradeoffs
- Case Ch→C combines channel parallelism with compact Key placement and, for filled Value blocks, preserves contiguous SV reductions; Case R reduces Value write row activations.A single static Value layout cannot both preserve long GEMV reductions and minimize row activations for single-token writes, so PATTON uses Case R while a block fills and Case Ch→C afterward.Source: §3.2.1 · pp. 4–5, Figure 4 · p. 5, §3.2.2 · p. 5
- Block-granularity sharing improves capacity, but splitting the same 2 KB MAC work into eight fragments raises latency 2.18× and energy 1.34× versus one continuous sequence.Reserving an entire 2 KB row before decoding finishes can leave columns unavailable to other requests and reduce capacity efficiency.Source: §3.2.2 · pp. 5–6
3.3 Limitations of Prior Works
- Prior systems leave gaps: some map large tensors or coarse chunks without specifying token-level placement, while others preallocate capacity or require processing-unit modifications.AttAcc’s initialization-time mapping limits dynamic block-level management; AttenPIM and L3 require functionality beyond commodity PIM’s MAC operation and lack dynamic allocation and PagedAttention support.Source: §3.3 · pp. 6–7
3.4 Design Requirements
- PATTON’s design requirements bind each logical block to exact physical coordinates, use distinct Key and Value layouts, balance Value write, GEMV, and capacity efficiency, and retain the existing PIM command set.A coarser allocation unit groups block-sized mappings so requests can share row capacity without frequently fragmenting SV.Source: §3.4 · p. 7
4 PATTON
PATTON maps serving engines’ logical KV blocks to physical placements and PIM commands while managing memory for attention execution. Its hierarchy groups block mappings for GEMV and capacity use, and its Commit Zone stages partial Value blocks for efficient writes.
4.1 Overview
- PATTON manages physical placements while vLLM retains request scheduling and logical KV-cache management, including prefix sharing and caching.Its hierarchy links block mapping, physical allocation, and command generation.Source: §4.1 · p. 7
4.2 Hierarchical Granule Allocation
- Order-1 granules bind each logical Key or Value block to a physical placement, while Order-2 granules group blocks for GEMV efficiency and capacity use.Order-2 allocation packs mappings into fewer rows while preserving useful GEMV spans and allowing requests to share row sets.Source: §4.2 · pp. 7–8, Figure 5 · p. 8
4.3 Key Cache Mapping and Execution
- For QK⊤, PATTON maps Key blocks across banks and duplicates the Query at O1K column offsets before issuing MACAB and RDMAC commands.Each block retains its own accumulator initialization and result-retrieval boundary.Source: §4.3 · p. 8
4.4 Value Cache Mapping and Commit Zone
- The Commit Zone holds a request’s partial Value block until full, then transfers it to its reserved O1V destination for GEMV execution.Order-2-V groups adjacent O1Vs to extend the token reduction while leaving other groups available to other requests.Source: §4.4 · pp. 8–9
- 12.6× fewer row activations for single-token Value writes, including commit overhead, with the Commit Zone.Partial blocks are staged in CZ slots before moving to their GEMV-optimized O1V destinations.Source: Figure 6 · p. 9, §4.4 · p. 9
4.5 Mapping Metadata and Block Lifecycle
- PATTON tracks each block’s Key and visible Value placement, supports shared and cached full blocks through reference counts, and releases mappings on eviction.Partial Value blocks remain private and writable in CZ slots; terminating a request releases both its slot and reserved O1V destination.Source: §4.5 · p. 9
5 Evaluation
The evaluation tests PATTON under dynamic, multi-turn serving workloads and compares its runtime and PIM layout with translated baselines. PATTON leads the baseline comparisons, while ablations examine the Commit Zone and the balance between GEMV efficiency and cache capacity.
5.1 Dataset and Models
- The evaluation uses multi-turn ShareGPT conversations with sustained arrivals and departures, running each configuration for more than 10,000 vLLM scheduler steps.Target concurrency and live-pool sizes vary by model; comparisons with AttAcc and NeuPIMs use a 2,048-token context limit, which PATTON itself does not impose.Source: §5.1 · p. 10
5.2 Runtime Layouts
- Baselines use a common command-generation interface while retaining documented allocation semantics; all runtimes use the same 16 GiB AiM configuration.Where exact vector placements were unspecified, the evaluation applies compatible GEMV-optimized layouts; BlockPIM uses a hardware-bound 128-token block size.Source: §5.2 · p. 10
5.3 Performance and Energy Evaluation
- PATTON averaged 4.27×, 1.42×, 1.52×, and 1.57× speedups over NeuPIMs, PIMphony, AttAcc, and BlockPIM, with energy-efficiency improvements of 7.94×, 6.77×, 6.70×, and 1.51×, respectively.Figure 7 reports latency and energy across four models, normalized to PATTON.Source: §5.3 · p. 10, Figure 7 · p. 11
5.4 Ablation Studies
- Against PATTON-wo-CZ, PATTON cuts overall attention latency by 13.0% and energy by 12.4%, while accelerating Value writes by 5×.Figure 8 compares BlockPIM, PATTON-wo-CZ, and PATTON on Qwen1.5-7B.Source: Figure 8 · p. 11, §5.4 · p. 11
- Under capacity pressure, KV cache hit rates are 87.86% for GPU_cache, 82.09% for PATTON, and 30.30% for PATTON without O2V; without pressure, all reach 91.28%.Whole-VRS allocation without O2V attains the highest raw GEMV efficiency but strands columns, reducing hit rate and causing additional GPU prefill recomputation; O2V avoids this while preserving contiguous SV execution.Source: Figure 9 · p. 11, §5.4 · pp. 11–12
5.3 Performance and Energy Evaluation
- PATTON's gains combine efficient GEMV execution and decode-stage KV writes; fine-grained allocation and block-level cache management also nearly eliminate runtime-induced prefill recomputation against NeuPIMs, PIMphony, and AttAcc.Against BlockPIM, PATTON improves KV writes, sustains channel parallelism during dynamic request activity, and preserves contiguous Value placement for SV reduction.Source: §5.3 · p. 11
6 Conclusion
PATTON translates serving engines’ block-level KV cache operations into explicit placements and commands for commodity PIM. Its allocation and Commit Zone mechanisms jointly support efficient execution and dynamic cache management without PIM processing-unit modifications.
6 Conclusion
- PATTON uses Order-1 granules to bind logical token blocks to physical placements and commands, and Order-2 granules to group mappings for GEMV efficiency and capacity use.The Commit Zone improves decode-stage KV cache writes.Source: §6 · p. 12
- PATTON supports dynamic allocation, prefix sharing, prefix caching, reclamation, and attention execution without modifying PIM processing units.The runtime combines GEMV, KV cache write, and capacity efficiency.Source: §6 · p. 12
- Across attention and runtime-induced prefill recomputation, PATTON accelerates the workload by 1.95× and improves energy efficiency by 4.83× on average over evaluated baselines.Source: §6 · p. 12