Source-linked AI summary
Challenges and Research Directions for Large Language Model Inference Hardware
Xiaoyu Ma, David Patterson
TL;DR
LLM Decode inference is difficult because autoregressive generation makes it memory bound, while recent model trends increase memory, communication, and latency demands. The paper reviews these bottlenecks and proposes hardware directions centered on high-capacity and high-bandwidth memory, 3D integration, and low-latency interconnects. It focuses on datacenter inference while considering constrained adaptations for mobile devices.
Problem
Autoregressive Decode differs from training and exposes memory, bandwidth, and interconnect bottlenecks that mainstream training-oriented accelerators handle inefficiently.
Method
The paper reviews LLM inference phases and hardware bottlenecks, then proposes four architecture research directions for datacenter inference and discusses mobile applicability.
Results
The paper recommends HBF, PNM, 3D memory-logic stacking, and low-latency interconnect as directions for improving LLM inference hardware.
Takeaways & Limitations
LLM inference hardware should prioritize memory capacity, memory bandwidth, and network latency alongside datacenter capacity, power, cost, and carbon metrics.
Takeaways & Limitations
HBF write endurance limits its use for frequently changing data, while PNM versus PIM is less clear for mobile devices.
Abstract
from arXiv · showhide
Large Language Model (LLM) inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerbated by recent AI trends, the primary challenges are memory and interconnect rather than compute. To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication. While our focus is datacenter AI, we also review their applicability for mobile devices.
INTRODUCTION
LLM inference is increasingly costly as new model trends expand memory, communication, and latency demands. The paper motivates hardware innovation to address these growing challenges.
- The paper proposes research directions intended to address major hardware challenges facing AI industry practice.
- LLM inference costs threaten economic viability as model usage increases and projected inference-chip sales grow 4X-6% over five to eight years.
- MoE scales model size through selectively invoked experts but expands inference memory and communication requirements.
- Reasoning models generate long thought sequences that greatly increase latency and strain memory.
- Multimodal generation, longer context, and retrieval-augmented generation increase inference resource demands.
- Diffusion generates all tokens in one step and iteratively denoises them, increasing compute demands rather than the autoregressive bottlenecks emphasized here.
CURRENT LLM INFERENCE HARDWARE AND ITS INEFFICIENCIES
Transformer inference separates into compute-bound Prefill and memory-bound autoregressive Decode, making mainstream training-oriented accelerators inefficient for Decode. The resulting bottlenecks are memory capacity, bandwidth, latency, and interconnect communication.
- Inference phases: Prefill processes input tokens in parallel and is often compute bound, whereas autoregressive Decode generates one token per step and is memory bound.The KV Cache connects the phases and grows with input and output sequence length.
- Inference phases: Mainstream GPUs and TPUs were not designed solely for LLM inference, and their training-oriented designs are inefficient for Decode.Decode differs substantially from training because it is inherently sequential and memory bound.
- Memory: 80X growth in NVIDIA GPU 64-bit FLOPS from 2012 to 2022 contrasted with 17X bandwidth growth, widening the memory wall.
- Memory: HBM capacity and bandwidth costs each grew 1.35x from 2023-2025, while DDR4 capacity cost fell to 0.54x and bandwidth cost to 0.45x from 2022-2025.Manufacturing and packaging difficulties contribute to HBM cost increases, while DRAM density growth is also decelerating.
- Memory: SRAM-only designs were overwhelmed by LLM capacity requirements and later required external DRAM.
- Latency: Longer outputs, inputs, KV caches, RAG, and reasoning sequences increase time-to-completion or time-to-first-token.
- Interconnect: Multi-chip inference requires frequent communication, while small Decode messages make interconnect latency more important than bandwidth.MoE and long-sequence models can further increase system size and hop count.
- Research direction: The paper therefore focuses on memory and interconnect improvements rather than compute, identifying research opportunities to address Decode bottlenecks.
FOUR RESEARCH OPPORTUNITIES TO RE-THINK LLM INFERENCE HARDWARE
The paper proposes four hardware directions for LLM Decode inference: High Bandwidth Flash, Processing-Near-Memory, 3D memory-logic stacking, and low-latency interconnect. Together, these directions target memory capacity, bandwidth, communication latency, and system efficiency under datacenter constraints.
- The four directions are designed to improve performance/TCO, performance/CO2e, and performance/power for Decode inference.The paper presents them as synergistic opportunities that can be combined in one architecture.
- ① High Bandwidth Flash for 10X capacity: High Bandwidth Flash combines HBM-like bandwidth with flash capacity and provides 10X memory capacity per node.Its larger capacity can reduce system size, power, TCO, CO2e, and network overhead, while flash capacity continues to scale faster than DRAM.
- ① High Bandwidth Flash for 10X capacity: HBF is best suited to infrequently updated data because limited write endurance and page-based reads make small, frequent accesses inefficient.HBF cannot replace all HBM; systems still need conventional DRAM for data unsuitable for flash.
- ① High Bandwidth Flash for 10X capacity: HBF can host larger weights and slow-changing contexts, including giant MoEs, web corpora, code databases, and paper corpora.Its capacity can also downsize inference systems and reduce dependence on HBM-only architectures.
- ② Processing-Near-Memory for high bandwidth: PNM places memory and logic on nearby separate dies, whereas PIM places them on the same die, making the distinction operationally unambiguous.For datacenter LLM inference, PNM permits shards 1000x larger than PIM, simplifying partitioning and reducing communication overhead.
- ③ 3D memory-logic stacking for high bandwidth: 3D memory-logic stacking uses vertical TSVs for dense, low-power memory interfaces, while its main challenges are thermal management and memory-logic coupling.The paper also identifies software mapping, inter-stack communication, and bandwidth-power-thermal-reliability tradeoffs as open questions.
- ④ Low-latency interconnect: Higher memory bandwidth and capacity can reduce Decode latency and communication overhead, while datacenter interconnects should prioritize latency because Decode messages are small and communication is frequent.The paper points to high-connectivity topologies as one example of this direction.
RELATED WORK
The paper reviews hardware and software directions for LLM inference, including High Bandwidth Flash, Processing-Near-Memory, low-latency interconnects, and software-hardware codesign.
- High Bandwidth Flash: High Bandwidth Flash applies an HBM-like architecture to flash to overcome flash bandwidth limits and provide dense AI-inference memory.Related work also considers flash integrated with mobile processors, using LPDDR for Prefill and Processing-Near-Flash for Decode.
- Processing-Near-Memory: Processing-Near-Memory research includes 3D compute-logic stacking, commercial DDR-attached processors, and CXL-based designs for programmability and system integration.Examples include compute-on-HBM-base-die proposals, Samsung AXDIMM, and Marvell Structera-A.
- Low-latency interconnect: Low-latency interconnect research explores low-hop-count topologies and in-network capabilities such as reduction and multicast acceleration.Examples include trees, dragonfly, high-dimensional Tori, NVIDIA NVLink, InfiniBand switches, and SHARP.
- Software Innovations: Software-hardware codesign offers algorithmic and software alternatives for improving LLM inference, including non-autoregressive generation such as Diffusion for image generation.Avoiding autoregressive generation could simplify AI inference hardware.
CONCLUSION
LLM inference increasingly requires lower cost and latency, while autoregressive Decode creates memory and interconnect challenges worsened by newer model and workload trends. The paper recommends four hardware directions and broader evaluation tools and metrics to address this mismatch.
- Research Need: Autoregressive Decode creates major memory and interconnect-latency challenges, exacerbated by MoE, reasoning, multimodal data, RAG, and long sequences.The conclusion frames these challenges as an attractive research target because LLM inference needs lower cost and latency.
- Research Tools: A roofline-based performance simulator should estimate memory capacity, sharding techniques, and modern performance and cost metrics for LLM inference.The proposed framework is motivated by the architecture community’s prior progress when realistic simulators were available.
- Recommended Directions: The paper recommends High Bandwidth Flash, Processing-Near-Memory, 3D stacking, and low-latency interconnect instead of focusing primarily on compute.It also proposes metrics centered on datacenter capacity, system power, and carbon footprint, with constrained variants potentially applicable to mobile devices.
- Implication: The authors present these directions as opportunities for collaborative innovation toward more affordable AI inference.The stated goal is to support innovations needed for delivering affordable AI inference.
APPENDIX: DDR DRAM PRICE 1957-2024
The appendix presents historical DDR DRAM price records spanning multiple decades, with entries listing dates, capacities, costs, speeds, memory types, and vendors.
- Data Source: The appendix identifies its data source as a historical memory-price collection maintained by John C. McCallum.The source website is described as no longer available, and the collection was intended to be passed to a new maintainer.
- Record Format: The records use fields including date, price per gigabyte, kilobytes, U.S. dollars, nanoseconds, size, company, speed, and memory type.The appendix also labels reference page and company-related columns.
- Historical Coverage: The appendix includes historical memory entries from 1973 and 1976 alongside later DDR and SIMM records.Early entries include PDP8/e core memory and SD Sales static memory boards, while later records include 1996–1997 SIMM listings.
- DDR Listings: Later entries document DDR2, DDR3, and DDR3L modules with capacities from gigabytes to tens of gigabytes and prices from individual online listings.Listings identify dates, capacities, prices, timings, speeds, vendors, and memory technologies through 2015.