PaperlayerView exact revision on arXiv

Source-linked AI summary

PIM Is All You Need: A CXL-Enabled GPU-Free System for Large Language Model Inference

Yufeng Gu, Alireza Khadem, Sumanth Umesh, Ning Liang, Xavier Servot, Onur Mutlu, Ravi Iyer, Reetuparna Das

arXiv:2502.07578v3cs.AR

TL;DR

LLM inference is difficult to deploy efficiently because large parameters and KV caches demand memory capacity while low-operational-intensity decoding underutilizes GPU compute. CENT combines CXL memory expansion, hierarchical PIM-PNM processing, scalable CXL communication, and parallel mappings to provide GPU-free inference. Compared with GPU baselines, it reports higher throughput, lower energy use, and more tokens per dollar.

  • Problem

    Large LLM parameters and per-prompt KV caches require substantial memory, while low-operational-intensity inference leaves GPU compute resources underutilized.

  • Method

    CENT combines CXL memory expansion, hierarchical PIM-PNM processing, scalable CXL communication, and parallel mappings across CXL devices.

  • Results

    2.3× higher throughput, 2.3× less energy, and 5.2× more tokens per dollar are reported versus GPU baselines in the conclusion.

  • Takeaways & Limitations

    CENT offers greater cost-efficiency for LLM inference, with higher throughput speedups for long-context reasoning and video-generation workloads due to high memory bandwidth.

  • Takeaways & Limitations

    PIM has lower memory density, and CENT does not support batching within a single pipeline stage because it would increase buffer requirements and latency.

Abstract

from arXiv · show

Large Language Model (LLM) inference uses an autoregressive manner to generate one token at a time, which exhibits notably lower operational intensity compared to earlier Machine Learning (ML) models such as encoder-only transformers and Convolutional Neural Networks. At the same time, LLMs possess large parameter sizes and use key-value caches to store context information. Modern LLMs support context windows with up to 1 million tokens to generate versatile text, audio, and video content. A large key-value cache unique to each prompt requires a large memory capacity, limiting the inference batch size. Both low operational intensity and limited batch size necessitate a high memory bandwidth. However, contemporary hardware systems for ML model deployment, such as GPUs and TPUs, are primarily optimized for compute throughput. This mismatch challenges the efficient deployment of advanced LLMs and makes users pay for expensive compute resources that are poorly utilized for the memory-bound LLM inference tasks. We propose CENT, a CXL-ENabled GPU-Free sysTem for LLM inference, which harnesses CXL memory expansion capabilities to accommodate substantial LLM sizes, and utilizes near-bank processing units to deliver high memory bandwidth, eliminating the need for expensive GPUs. CENT exploits a scalable CXL network to support peer-to-peer and collective communication primitives across CXL devices. We implement various parallelism strategies to distribute LLMs across these devices. Compared to GPU baselines with maximum supported batch sizes and similar average power, CENT achieves 2.3$\times$ higher throughput and consumes 2.9$\times$ less energy. CENT enhances the Total Cost of Ownership (TCO), generating 5.2$\times$ more tokens per dollar than GPUs.

1 Introduction

LLM inference combines large memory demands with low operational intensity, leaving GPU compute underutilized. CENT addresses this mismatch with CXL memory expansion, hierarchical PIM-PNM processing, scalable communication, and parallel mappings across CXL devices.

  • Motivation: Large parameters and per-prompt KV caches require substantial memory, while low operational intensity leaves multi-GPU compute resources underutilized.Modern context windows range from 128K to 1M tokens, further increasing KV-cache capacity requirements.
  • CENT Architecture: CENT uses CXL memory expansion and hierarchical PIM-PNM processing to provide LLM capacity and execute transformer blocks without GPUs.Near-bank PUs handle most arithmetic operations, while PNM accelerators and RISC-V cores handle Softmax, square root, division, and other complex operations.
  • CXL Network and Parallelism: CENT supports peer-to-peer and collective communication primitives across CXL devices for distributing LLM computation.The system uses a CXL switch and shared buffers to support send/receive, broadcast, multicast, and gather operations.
  • CXL Network and Parallelism: Pipeline parallelism prioritizes throughput, tensor parallelism reduces latency, and hybrid TP-PP mappings balance the two objectives.Pipeline parallelism assigns transformer blocks across memory channels, while tensor parallelism distributes each block across CXL devices.
  • Evaluation: 2.3× higher throughput, 2.9× less energy, and 5.2× more tokens per dollar are reported versus GPU baselines at maximum supported batch sizes and similar average power.CENT is evaluated on Llama2 models and can provide greater cost-efficiency as model sizes and context lengths increase.

2 Motivation

Decoder-only LLM inference is constrained by growing memory requirements and low operational intensity, while PIM offers bandwidth but suffers from low density. CENT combines CXL expansion, scalable networking, and PIM-PNM mapping to address these constraints.

  • High Memory Capacity Requirement: Long context windows create large per-user KV caches that limit inference batch size and increase memory-capacity requirements.Modern context windows range from 128K to 1M tokens.
  • Low Operational Intensity: Decoding uses sequential GEMV operations with substantially lower operational intensity than the GEMM operations used during prefill.Batching can combine GEMV operations across queries into GEMM operations, improving operational intensity.
  • GPU Performance Characterization: 21% GPU compute utilization is reported for Llama2-70B, and decoding an output token takes 3.4× longer than encoding a prompt token.BERT and ResNet-152 predominantly use high-operational-intensity GEMM operations and better utilize GPU compute throughput.
  • PIM Provides Higher Memory Bandwidth: 16 TB/s internal bandwidth for GDDR6-based AiM is compared with 2 TB/s external bandwidth for an A100 GPU with five HBM2E stacks.PIM's internal bandwidth is presented as suitable for low-operational-intensity LLM inference.
  • Low Memory Density of PIM: PIM memory density falls to 25%−50% for UPMEM R-DIMM and 75% for AiM compared with conventional memory.These density reductions create challenges for scaling PIM to LLMs with substantial memory demands.
  • Scalable Network of PIM: CXL 3.0 provides a scalable interconnect for PIM devices, with CXL.mem offering ∼8× lower latency than network-based RDMA.The design also requires collective communication primitives and parallelization strategies to map LLMs across devices.
  • Hierarchical PIM-PNM Architecture: CENT uses hierarchical PIM-PNM mapping, assigning MAC-heavy operations to near-bank PUs and other transformer operations to shared PNM units.This design addresses the memory-density overhead of general-purpose near-bank processing.

3 Background

Decoder-only LLM inference first processes prompts in parallel during prefill and then generates tokens sequentially during decoding. Its computation is organized into repeated decoder blocks containing self-attention, feed-forward, residual, and normalization components.

  • Inference Stages: The prefill stage encodes prompt tokens in parallel, whereas the decoding stage generates output tokens sequentially.Both stages include input embedding, decoder transformer blocks, output embedding, and sampling layers.
  • Decoder Transformer Blocks: Decoder transformer blocks contain self-attention and feed-forward network layers paired with residual connections and normalization layers.The Llama2 architecture is presented as a representative decoder-only LLM.
  • Self-Attention: In self-attention, query, key, and value vectors are generated from the input vector using corresponding weight matrices.The matrices are segmented into multiple heads representing different semantic dimensions.
  • Self-Attention: RoPE encodes relative positional information, after which key and value vectors are appended to their caches for attention computation.The query is multiplied by the key cache to produce a score vector before Softmax and value-cache multiplication.

4 CENT Architecture

CENT combines a CXL network, hierarchical PIM-PNM hardware, and a dedicated instruction set to execute LLM transformer blocks without GPUs.

  • CXL-based Network Architecture: CENT connects 32 CXL devices through a CXL switch driven by a host CPU.Each device integrates a CXL controller, PNM units, and 16 memory chips with two GDDR6-PIM channels per chip.
  • CXL-based Network Architecture: The CXL network supports host-to-device and peer-to-peer communication, including broadcast, send, receive, and collective operations.Broadcast is implemented through reserved header encoding and routing at the CXL switch.
  • Hierarchical PIM-PNM Architecture: Each GDDR6-PIM channel combines a Global Buffer, bank-local processing units, and DRAM capacity for parallel in-memory computation.Each bank provides 32MB and includes a near-bank PU; the Global Buffer broadcasts 256-bit data to PUs concurrently.
  • Hierarchical PIM-PNM Architecture: PNM units provide accumulators, reduction trees, exponent accelerators, and RISC-V cores for operations beyond MACs.A 64KB Shared Buffer and data-movement instructions connect PIM channels with PNM units.
  • ISA Summary: CENT arithmetic and data-movement instructions specify PIM channels, registers, activation functions, and DRAM or Shared Buffer locations.The ISA decodes instructions into micro-operations directed to PIM controllers and PNM units.

5 Model Mapping

CENT maps LLM computation across CXL devices using pipeline, tensor, and hybrid parallelism, with fine-grained PIM-PNM placement for transformer operations.

  • Pipeline-Parallel Mapping: Pipeline parallelism assigns transformer blocks to stages and processes multiple prompts concurrently across different stages.Stages are mapped to CXL devices and use intra-device or peer-to-peer transfers between stages.
  • Pipeline-Parallel Mapping: CENT does not batch queries within a pipeline stage because buffering multiple embeddings requires larger Global and Shared Buffers.The paper also states that batching on top of pipeline parallelism would increase latency because pipeline stages already fully use PIM resources.
  • Tensor-Parallel Mapping: Tensor parallelism distributes each transformer block across all CXL devices, targeting lower inference latency.Fully connected layers are distributed, while residual and normalization layers remain on a master device; attention distribution requires AllReduce.
  • Hybrid Tensor-Pipeline Parallel Mapping: Hybrid TP-PP assigns each decoder to multiple consecutive devices, combining tensor parallelism for latency with pipeline parallelism for throughput.For 32 devices, mapping each decoder to 8 devices yields TP=8 and PP=4.
  • Transformer Block Mapping: Fine-grained mapping executes complete transformer blocks within CXL devices, assigning operations among PIM channels, PNM accelerators, and RISC-V cores.PIM handles operations such as GEMV and vector dot products, while PNM cores support rotary embedding transformations.
  • End-to-End Execution: CENT supports end-to-end inference by mapping prefill and decoding transformer computation to CXL devices and performing top-k sampling on the host CPU.The prefill stage fills KV caches token by token, while decoding generates output tokens sequentially.

6 Methodology

The evaluation compares CENT with GPU systems using matched model workloads, power assumptions, hardware-cost estimates, and three-year ownership costs.

  • System Configuration: CENT is compared with a baseline containing 4 NVIDIA A100 80GB GPUs connected by NVLink 3.0.The CENT configuration uses 32 CXL devices and is designed to have similar average power to the GPU system.
  • Workloads: The benchmark evaluates Llama2 7B, 13B, and 70B models with 512 prefill tokens and 3584 decoding tokens per query.The resulting 4K context length matches the maximum supported by the evaluated Llama2 models.
  • Power Modeling: Power evaluation models DRAM, MAC, memory-controller, and RISC-V-core power using device specifications and stated assumptions.The remaining CXL-controller components are implemented in RTL and synthesized using a TSMC 28nm technology library.
  • Hardware Cost Modeling: The estimated CXL-controller area is 19.0mm^2 in 7nm after scaling custom logic and incorporating controller, PCIe, and PHY measurements.The estimate combines synthesized custom logic with measurements from NVIDIA GPU die shots.
  • TCO Modeling: Three-year TCO includes owned and rental hardware costs, host CPU pricing, CXL-device cost estimates, electricity, and average power.Rental CXL-device costs are estimated using the owned-TCO methodology because reference rental prices were unavailable.
  • Hardware Cost Modeling: CXL-controller cost is decomposed into die, packaging, and non-recurring engineering components.The die-cost estimate uses wafer cost, area, and yield; packaging and production-volume assumptions determine additional components.

7 Results

CENT outperforms GPU baselines across throughput, latency, cost efficiency, and energy efficiency, particularly during memory-bound decoding and long-context inference. Its CXL-connected PIM architecture also scales across devices and provides a GPU-free alternative to heterogeneous GPU-PIM systems.

  • CENT versus GPU Baseline: 4.6× lower end-to-end latency than GPUs is achieved in the latency-critical batch-size-1 scenario.CENT uses tensor parallel mapping for this case, with the speedup attributed to higher internal PIM memory bandwidth.
  • CENT versus GPU Baseline: 2.5× higher decoding throughput is achieved by CENT, while GPUs provide 2.5× higher prefill throughput.Prefill accounts for only 2% of total GPU end-to-end processing time, so overall throughput closely follows decoding performance.
  • CENT versus GPU Baseline: 3.4–7.6× lower query latency is achieved by CENT at similar throughput to the baseline GPU.The comparison uses different GPU batch sizes and different tensor- or pipeline-parallel mappings on CENT.
  • Power and Energy Consumption Analysis: 2.9× more tokens per Joule are processed by CENT on average than by GPUs.GPU is 2.4× more energy efficient in compute-bound prefill, whereas CENT achieves 3.2× higher energy efficiency in memory-bound decoding.
  • CENT versus PIM/PNM Baselines: 4.5× higher throughput is achieved by CENT than CXL-PNM at the maximum supported batch sizes.CENT provides higher compute throughput and memory bandwidth, but less memory capacity.
  • CENT versus PIM/PNM Baselines: 1.8–3.7× and 1.8–5.3× more tokens per dollar are achieved by CENT than AttAcc and NeuPIM, respectively.Across the evaluated sequence lengths and batch sizes, CENT raw throughput ranges from 0.5–1.1× AttAcc and 0.7–2.1× NeuPIM.

8 Related Work

Related work spans ML accelerators, CXL memory expansion, PIM and PNM systems, and transformer-specific accelerators. Prior systems also explore near-data processing in DIMMs and CXL switches for specialized workloads.

  • Related Work: Recent research explores ML accelerators, hardware/software co-designs, CXL memory expansion, PIM, PNM, and transformer accelerators.Transformer accelerator examples include systems targeting encoder inference such as BERT.
  • Related Work: Beacon investigates near-data processing in DIMMs and CXL switches for genome sequencing analysis.Its processing units are customized for that specialized workload.

9 Conclusion

The paper concludes that CENT combines PIM’s high internal memory bandwidth with CXL memory expansion to address decoder-only LLM inference requirements. Against GPU baselines, it reports higher throughput, lower energy consumption, and greater tokens per dollar.

  • Conclusion: 2.3× higher throughput, 2.3× less energy, and 5.2× more tokens per dollar than GPU baselines are reported for CENT.The comparison uses GPU baselines with maximum supported batch sizes.

A.1 Abstract

The reproduction guide describes how to regenerate the paper’s main performance, power, cost-efficiency, and energy-efficiency figures. It covers the simulator, supporting scripts, packages, execution, result processing, and figure generation.

  • A.1 Abstract: The guide covers reproducing the main performance, power, cost-efficiency, and energy-efficiency results in Figures 12–15.It includes repository cloning, simulator building, package setup, end-to-end simulation, result processing, and figure generation.

A.2 Artifact check-list (meta-information)

The artifact supports reproducing Figures 12–15 through trace generation and simulation, result processing, power modeling, and figure generation. It documents the required software, models, commands, and resource settings for the workflow.

  • Models and experiments: The workflow evaluates Llama2 7B, 13B, and 70B models using PIM trace generation, simulation, and CENT power modeling.Model weights are required only for functional simulation, while the performance simulator and power model do not model real values.
  • Artifact contents: The artifact includes a trace generator, AiM simulator, power model, figure generator, and automation script for reproducing Figures 12–15.The figures also use expected results from a four-Nvidia-A100-80GB-GPU baseline.
  • Software requirements: The artifact requires C++ compilation with g++-11/12/13 or clang++-15 and Python packages including pandas, matplotlib, torch, and scipy.The artifact provides setup and build commands for the AiM simulator and Python environment.
  • Simulation workflow: The simulation pipeline generates traces, processes simulation logs, calculates latencies, and estimates energy consumption and average power.Generated traces and logs are stored in the trace directory, while processed latency and power results are written to cent_simulation/simulation_results.csv.
  • Resource requirements: Sequential token simulation requires approximately 100GB of disk space and about 24 hours with 8 threads or 12 hours with 96 threads.Setting SEQ_GAP to 128 simulates one out of every 128 tokens to reduce disk usage and simulation time.
  • Result processing: The results-processing stage computes latency, throughput, power, and energy for prefill, decoding, and end-to-end phases.Processed results are stored in cent_simulation/processed_results.csv.
Loading 2502.07578v3…