PaperlayerView exact revision on arXiv

Source-linked AI summary

AI Hardware Accelerators for Large Language Models: Architectures and the Memory Wall

Siddharth Patel, Rohit Singh

arXiv:2608.28048v1cs.ARcs.DC

TL;DR

LLMs create growing hardware demands, and this review examines whether existing accelerator paradigms can meet them across workloads and deployment settings. Using Transformer workload analysis and roofline-based comparison, it finds that memory constraints dominate acceleration, favoring a heterogeneous toolkit in which GPUs, ASICs, and emerging memory-centric designs serve different roles. The review concludes that future progress depends on hardware–algorithm co-design and heterogeneous, memory-centric systems.

  • Problem

    LLM acceleration is constrained by bandwidth-bound decode, large KV caches, and data-movement costs, while accelerator research spans fragmented architectural and deployment contexts.

  • Method

    The review surveys accelerator families across cloud training, cloud inference, and edge inference using Transformer workload requirements, roofline analysis, and cross-paradigm comparisons.

  • Results

    No single architecture is optimal: GPUs remain flexible training workhorses, ASICs win at stable scale, and PIM is the most promising near-term memory-wall response.

  • Takeaways & Limitations

    LLM acceleration should combine heterogeneous hardware with quantization, sparsity, attention redesign, and memory-system co-design rather than seek one GPU successor.

  • Takeaways & Limitations

    PIM and near-memory systems face DRAM, density, consistency, programmability, and data-layout constraints, limiting their role to heterogeneous complements rather than GPU replacements.

Abstract

from arXiv · show

Large language models (LLMs) place unprecedented and still-growing demands on the hardware that trains and serves them. This review surveys the full landscape of AI hardware accelerators for LLMs, including general-purpose GPUs, custom ASICs such as TPUs, Trainium, Groq, and Cerebras, reconfigurable FPGAs, processing-in-memory and near-memory architectures, and emerging neuromorphic and photonic approaches across cloud and edge deployment. Using the transformer's computational structure and roofline analysis as a common framework, we show that the decisive constraint on LLM acceleration is not arithmetic but memory: the autoregressive decode phase is bandwidth-bound, the key-value cache can rival the model weights in size, and data movement dominates energy. Comparing platforms on compute, memory, energy, programmability, and scalability, we find that no single architecture is optimal across workloads: GPUs remain the flexible default and the workhorse of training; domain-specific ASICs win at scale for stable, high-volume workloads; processing-in-memory is the most promising near-term response to the memory wall, entering systems as a heterogeneous complement; and neuromorphic and photonic computing, while promising, are not yet production-ready at frontier scale. Future progress depends on hardware-algorithm co-design and heterogeneous, memory-centric systems: for large language models, the memory system has become the computer.

1 Introduction

LLMs have rapidly expanded hardware demands, while existing accelerator surveys leave the broader landscape without a unified, workload-grounded framework. This review argues that memory bandwidth, capacity, and data movement—not arithmetic alone—shape acceleration and motivate heterogeneous architectures.

  • The Large Language Model Revolution: Transformer-based LLMs replaced sequential recurrence with parallelizable self-attention, enabling unprecedented training scale and increasingly capable models.
  • The Hardware and Energy Crisis: LLM growth carries substantial resource costs, including an estimated 1,287 MWh of electricity and 552 tonnes of CO2-equivalent for GPT-3 training.
  • The Hardware and Energy Crisis: Off-chip DRAM access costs roughly 1,300–2,600 pJ per 64-bit access, versus only a few picojoules for an on-chip floating-point operation.This energy gap is roughly three orders of magnitude.
  • The Hardware and Energy Crisis: Autoregressive decode repeatedly streams weights and the growing KV cache, making LLM serving primarily a memory-bandwidth and data-movement problem.
  • Survey Scope: The survey covers GPUs, ASICs, FPGAs, PIM, near-memory, neuromorphic, and photonic accelerators across cloud training, cloud inference, and edge inference.
  • Why a Hardware Review Is Needed Now: Its unified taxonomy, workload-derived requirements, roofline analysis, cross-paradigm comparison, and open-problem discussion address fragmented prior coverage without claiming GPU displacement.

2 Background: LLM Architecture and Hardware Requirements

Decoder-only Transformers divide inference into compute-bound prefill and memory-bound decode, making memory bandwidth and data movement central hardware constraints. The review derives requirements spanning fast memory, low-precision and sparsity support, and co-designed architectures.

  • Transformer architecture: Decoder-only LLMs pair multi-head attention with a parameter- and FLOP-heavy feed-forward network, connected by residual paths and normalization.
  • Transformer architecture: Attention’s QK^T product creates an n × n score matrix, giving sequence-length cost O(n^2) that dominates latency and memory at very long contexts.
  • Transformer architecture: MQA and GQA reduce KV-cache size by sharing key–value projections across query heads, with GQA preserving most MHA quality and becoming common in major model families.
  • Inference profile: Prefill is compute-bound, whereas autoregressive decode is memory-bound because each generated token streams weights and the growing KV cache while reusing loaded weights minimally.
  • Inference profile: At long contexts, the KV cache rivals or exceeds model weights and, because it is reread every decode step, sets the generation bandwidth ceiling.
  • Hardware requirements: The roofline model bounds throughput by min(Ppeak, B · I), classifying low-intensity decode as bandwidth-bound and prefill or large-batch FFN work as compute-bound.
  • Hardware requirements: Hardware requirements therefore prioritize sustained bandwidth, nearby large memory, GEMM engines, low-precision arithmetic, sparsity support, and high-bandwidth interconnect.
  • Quantization and sparsity: Quantization reduces memory footprint, bandwidth pressure, and compute energy, while structured sparsity maps predictably to hardware but irregular sparsity and MoE routing require co-design.

3 A Taxonomy of LLM Hardware Accelerators

The review organizes accelerators by architecture, deployment context, and optimization target because workload constraints and useful metrics vary across settings. Accelerator selection is therefore a workload-alignment problem rather than a search for one universally best device.

  • Organizing framework: The taxonomy classifies LLM accelerators along three orthogonal axes: architecture, deployment context, and optimization target.
  • By architecture: The architectural spectrum trades programmability against efficiency, from broadly capable GPUs and CPUs to narrower, more efficient domain-specific accelerators.
  • By deployment context: Cloud training emphasizes aggregate throughput and high-bandwidth interconnect, whereas cloud inference emphasizes cost per token and latency under serving loads.
  • By deployment context: Edge inference is bounded by power, area, and thermal envelopes and is motivated by on-device privacy and offline operation.
  • By optimization target: Deployment objectives include throughput, latency, energy efficiency, and cost efficiency, which generally trade off against one another.
  • Cross-axis selection: Because the same architecture can serve different roles and the axes interact, the appropriate accelerator is the one aligned with the dominant metric and use case.

4 GPU-Based Acceleration

GPUs remain the default LLM platform because their architecture and software ecosystem support dense Transformer computation and deployment. However, decode is bandwidth-bound, limiting GPU utilization and motivating software and serving-level adaptations.

  • Why GPUs Became the Default: CUDA’s mature libraries, compilers, and frameworks preserve GPU incumbency despite competing platforms matching or exceeding some raw specifications.Tensor Cores and HBM support dense matrix multiplication, while the software ecosystem lowers deployment barriers and creates switching costs.
  • NVIDIA Architecture Evolution: NVIDIA’s accelerator generations increasingly emphasize memory capacity and bandwidth alongside Tensor Core throughput for evolving LLM workloads.Table 3 presents representative data-center GPU specifications, with dense BF16 figures used for comparability.
  • NVIDIA Architecture Evolution: Dense and sparse throughput must be distinguished because vendor sparse figures can be roughly twice dense performance.The review reports dense values throughout to maintain comparability across platforms.
  • GPU Bottlenecks for LLM Inference: Decode streams weights and the KV cache from HBM, leaving Tensor Cores underused; FlashAttention-2 reaches about 35% H100 utilization versus 80–90% for tuned matrix multiplication.Small batches worsen the problem through limited weight reuse, while multi-device communication adds further overhead.
  • Software and Serving Optimizations: IO-aware FlashAttention kernels improve GPU efficiency by tiling and fusing attention without materializing the full attention matrix in HBM.FlashAttention-2 reaches 50–73% of peak on A100, compared with 25–40% for the original version.
  • Software and Serving Optimizations: Disaggregated serving separates compute-bound prefill from bandwidth-bound decode across device pools, improving throughput per GPU and latency-constrained goodput.The KV cache is transferred between pools over high-bandwidth interconnects.
  • AMD and Other Competitors: AMD and Intel demonstrate specification parity without equivalent displacement because realized performance and software maturity remain decisive.MI300X reached about 620 BF16 TFLOPS against its 1,307 marketed figure in independent benchmarking.

5 Custom ASIC Accelerators

Custom ASICs divide between throughput-oriented designs and inference-oriented designs, but their shared objective is to reduce memory movement through specialized capacity, bandwidth, or locality. Their benefits come with trade-offs in flexibility, scale, and software maturity.

  • Google TPU Family: TPUs use systolic-array matrix multiplication and have evolved toward greater memory capacity, bandwidth, and inference efficiency across generations.The TPU family spans training-oriented and inference-oriented configurations, including large-scale pods and superpods.
  • Groq LPU: Groq uses deterministic compiler-scheduled execution and on-chip SRAM to trade capacity and chip count for predictable, low-latency inference.Serving Llama-2/3 70B requires roughly 576 LPUs across about nine racks and delivers about 300 tokens per second per user.
  • Cerebras WSE: Cerebras keeps an entire model on a wafer-scale device, using 44 GB of on-chip SRAM and 21 PB/s aggregate bandwidth to eliminate chip-to-chip communication.The WSE-3 contains over four trillion transistors and 900,000 cores; its published 125 PFLOPS figure is sparse FP16.
  • AWS ASICs: Trainium and Inferentia apply Amazon’s vertical-integration strategy to training and inference, combining specialized compute with substantial HBM capacity and bandwidth.Trainium2 delivers 667 dense BF16 TFLOPS with 96 GB of HBM3e at 2.9 TB/s.
  • Tenstorrent: Tenstorrent combines Tensix cores, local SRAM, and an open-source software stack, but verified model support and serving tools remain concentrated on older hardware.As of early 2026, FlashAttention-3 has no port and there is no production equivalent to vLLM’s continuous-batching API.
  • Qualcomm AI200 and AI250: Qualcomm’s AI200 prioritizes inference capacity with 768 GB of LPDDR per card, while AI250 targets over ten times the effective bandwidth through near-memory computing.The design accepts lower bandwidth than HBM in exchange for substantially greater capacity and lower cost.
  • Cross-Platform Comparison: Across custom ASICs, TPU, Trainium, and WSE favor training throughput, whereas Groq, Qualcomm, and Inferentia target inference latency or cost.All are presented as attempts to attack the memory bottleneck through SRAM, HBM, or computation closer to memory.

6 FPGA-Based Acceleration

FPGAs occupy a reconfigurable middle ground, allowing custom precision and dataflow to track changing LLM architectures. Their practical advantage is strongest in low-batch, latency-sensitive, and edge deployments, where lower throughput can be acceptable.

  • FPGA Architecture and Flexibility: FPGAs can be re-specialized after fabrication, making them adaptable to changing attention, mixture-of-experts, and state-space-model datapaths.DSP blocks support quantized matrix multiplication, while on-chip memory can keep activations close to compute.
  • Production and LLM Deployments: Project Brainwave demonstrated production-scale FPGA acceleration for real-time deep-neural-network serving through distributed model parallelism and low-latency hardware microservices.The program served Bing intelligent search and Azure workloads.
  • Precision and Sparsity Co-Design: INT4, INT8, block-floating-point, and configurable sparse datapaths let FPGAs co-match numeric formats and sparsity patterns to available silicon.MSFP achieved accuracy comparable to BF16 and INT8 at roughly 3× and 4× lower cost, respectively.
  • Limitations and Deployment Niche: FPGAs remain constrained by memory bandwidth: an HBM-equipped Alveo U280 provides about 460 GB/s, below contemporary GPU multi-terabyte-per-second bandwidth.This limits large-batch cloud serving and training throughput, while high-level synthesis and place-and-route increase engineering cost.
  • Limitations and Deployment Niche: The realistic FPGA niche is edge or embedded inference and latency-critical, low-batch, single-user serving where determinism and energy efficiency outweigh peak throughput.Within this niche, FPGAs complement rather than replace GPUs and ASICs.

7 Processing-in-Memory and Near-Memory Computing

Processing-in-memory and near-memory architectures address the memory wall by moving computation toward data, especially for bandwidth-bound attention and KV-cache operations. Current evidence favors heterogeneous systems that pair PIM with conventional processors rather than replacing them.

  • Motivation and Taxonomy: PIM attacks the memory wall by placing computation inside or near memory, reducing the energy and bandwidth costs of separating data and compute.In-DRAM PIM exploits internal bank bandwidth, while near-memory designs place logic on a base die, controller, or buffer.
  • Industry PIM Chips: Samsung HBM-PIM exposes roughly 4.92 TB/s of in-DRAM compute bandwidth versus 1.23 TB/s externally, a 4× internal advantage.The device is JEDEC-compatible and supports unmodified TensorFlow and PyTorch code.
  • Academic PIM Architectures for LLMs: Attention and KV-cache operations are strong PIM targets because they have low operations-per-byte ratios, while feed-forward GEMMs remain better suited to conventional processors.The KV cache can exceed model weights, increasing the importance of memory-centric execution.
  • Academic PIM Architectures for LLMs: AttAcc improved performance by up to 2.81× and energy efficiency by up to 2.67× for a 175B model over a same-capacity conventional system.Its heterogeneous design places attention on HBM-based PIM while retaining conventional processing for other workloads.
  • Heterogeneous Partitioning: A representative heterogeneous partition assigns compute-bound GEMMs to a GPU or NPU and memory-bound attention and KV-cache GEMV to HBM-PIM.This division is exemplified by AttAcc and NeuPIMs.
  • Challenges and Open Problems: PIM faces constraints from DRAM timing, reduced memory density, consistency management, limited programmability, bespoke software, and sensitivity to data layout.These constraints support using PIM as a heterogeneous complement rather than a GPU replacement.

8 Neuromorphic Chips for LLM Workloads

Neuromorphic processors co-locate memory and computation and use event-driven spikes to target efficient sparse workloads. Transformer demonstrations show feasibility, but current systems remain far below frontier LLM scale and are mainly suited to small edge models.

  • Architecture and Motivation: Neuromorphic processors use asynchronous spikes and co-located neuron state and synaptic weights, eliminating the data-movement bottleneck targeted by other architectures.Their efficiency depends on expending energy primarily when neurons emit spikes.
  • Representative Hardware: TrueNorth integrated one million neurons and 256 million synapses at roughly 65 mW, while Loihi 2 added programmable neuron models, graded spikes, and asynchronous communication.These devices established low-power operation and greater programmability as major neuromorphic directions.
  • Neuromorphic Approaches for Transformers: ANN-to-SNN conversion preserves accuracy but requires many simulation timesteps, whereas direct surrogate-gradient training avoids that conversion route.Spikformer removes softmax from spiking self-attention, and SpikeGPT reports roughly 5× lower energy on neuromorphic hardware.
  • Research and Commercial Activity: The spiking-language-model literature remains small relative to vision, despite demonstrations from Spikformer through SpikeGPT and larger neuromorphic systems such as Hala Point.These results establish feasibility rather than production-scale capability.
  • Honest Assessment: No current neuromorphic chip can run a frontier-scale LLM, and the largest demonstrated spiking language models are two to three orders of magnitude smaller than deployed models.Near-term use is therefore concentrated on edge inference of small or distilled models.
  • Honest Assessment: Neuromorphic computing is presently a promising research direction rather than a competitive platform for mainstream LLM inference.Its longer-term potential depends on scalable spiking-model training and improved software maturity.

9 Photonic and Optical Computing

Photonic computing can execute neural linear algebra with high bandwidth and potentially low energy, making it attractive for GEMM-heavy transformer phases. Precision, conversion overhead, scalability, and electronic memory requirements nevertheless keep photonic compute in the research stage, while photonic interconnect is the nearer-term opportunity.

  • Photonic Architectures: Photonic integrated circuits use light for neural linear algebra through MZI meshes, microring weight banks, and photonic crossbar arrays.These architectures target the massively connected linear layers of neural networks.
  • LLM Workload Fit: Optical multiplication is attractive for GEMM-heavy transformers because matrix multiplication dominates prefill and feed-forward layers, while wavelength multiplexing can increase throughput.The operation occurs as light propagates through an encoded-weight optical mesh.
  • Research Progress: Integrated photonics demonstrated trillion-scale multiply-accumulate rates through photonic tensor cores and on-chip silicon MZI networks.These demonstrations used optical frequency combs, phase-change-material weights, or externally configured matrix elements.
  • Challenges: Photonic matrix-vector multiplication has limited numerical precision, and reaching 16-bit precision requires digital-analog hybrid architectures that reintroduce digitization.Nonlinear activation functions also typically require conversion back to electronics.
  • Challenges: Electronic-optical interfaces consume much of the energy and area, while thermal resonance drift, parameter scalability, and absent optical memory add further constraints.Weights and KV cache must still reside in electronic memory, preserving the electronic memory wall.
  • Realistic Near-Term Role: For frontier LLMs, photonic computation remains in the research stage, while co-packaged optics and photonic interconnects offer a nearer-term way to relieve data-movement bottlenecks.The review identifies moving data, rather than replacing GPU datapaths, as the bankable photonic opportunity today.

10 Edge and Mobile LLM Deployment

Edge LLM deployment operates under much tighter power, memory, and bandwidth constraints than datacenters, making decode especially memory-bound. Quantized small models and hybrid on-device/cloud execution have made practical deployment possible, while frontier-scale models remain datacenter workloads.

  • Edge Constraints: Phones and laptops provide roughly one to ten watts, single-digit gigabytes of shared memory, and LPDDR bandwidth one to two orders of magnitude below datacenter HBM.These constraints make edge decode more severely memory-bound than datacenter inference.
  • Mobile Hardware: Mobile SoCs integrate dedicated NPUs, but vendor-stated TOPS figures are measured under differing conditions and are only rough capability classes.Apple’s A18 Pro Neural Engine is rated at 35 trillion operations per second.
  • Model and Software Techniques: Aggressive low-bit quantization brings multi-billion-parameter models within mobile memory budgets, with 4-bit weights serving as the practical workhorse.GPTQ, AWQ, and newer rotation-based methods such as SpinQuant reduce memory requirements while addressing low-bit accuracy loss.
  • The Edge Memory Wall: Edge deployment remains constrained by shared RAM, growing KV caches, and LPDDR bandwidth, which sets token-generation rate more than arithmetic throughput.Mitigations include weight and KV-cache quantization, plus edge-specific offloading strategies.
  • Use Cases and Outlook: Small on-device models handle latency- and privacy-sensitive tasks, while harder queries escalate to cloud models in a hybrid division of labor.Apple’s Private Cloud Compute exemplifies this pairing of on-device and server-based models.
  • Use Cases and Outlook: Edge silicon has made small-model inference ubiquitous, but frontier-scale models remain datacenter workloads for the foreseeable future.The current edge trajectory is centered on models with a few billion parameters.

11 Comparative Analysis

LLM accelerator fitness depends on workload-specific trade-offs among compute, memory, energy, software maturity, and scalability. Roofline and bandwidth analyses show decode is memory-bound, while GPUs, ASICs, FPGAs, and SRAM-resident designs occupy distinct operating points.

  • Six dimensions—compute throughput, memory capacity, bandwidth, energy efficiency, software maturity, and scalability—jointly determine accelerator fitness, with workload-dependent weights.
  • Decode is bandwidth-bound, whereas prefill is compute-bound and can approach peak throughput; increasing batch size shifts decode toward higher arithmetic intensity.
  • Tmax ≈ B/Smodel gives a first-order batch-1 decode ceiling in tokens per second when each token streams the model weights once.
  • 4-bit weight quantization raises the bandwidth ceiling fourfold, while KV-cache reads, attention, scheduling, and communication reduce measured throughput below the idealized bound.
  • Peak BF16 compute grew roughly seven-fold from A100 to B200, while HBM bandwidth grew about four-fold, indicating compute is outpacing bandwidth for memory-bound work.
  • DRAM access costs roughly two orders of magnitude more energy per byte than on-chip SRAM, making bandwidth hierarchy central to both performance and efficiency.
  • Verified workload figures are not mutually comparable because they use different models, batch sizes, and operations; specialization improves targeted operations rather than uniformly accelerating end-to-end serving.
  • GPUs remain the default for training and flexible inference; domain ASICs win at stable, high-volume scale, while FPGAs serve latency-sensitive and streaming niches.

12 Open Challenges and Future Directions

Open challenges center on memory movement, software maturity, energy measurement, and hardware–algorithm co-design rather than arithmetic alone. The review points toward heterogeneous systems and workload-representative benchmarking as practical directions.

  • Memory-centric architectures: Memory bandwidth, capacity, and data-movement energy govern LLM acceleration more than arithmetic does.Decode is bandwidth-bound, the key-value cache can rival the model in size, and DRAM access costs roughly two orders of magnitude more energy than on-chip access.
  • Software and programmability: Software maturity is the largest practical barrier to adopting non-GPU accelerators.CUDA and its surrounding libraries remain a durable moat, while alternatives are constrained by immature compilers, kernels, and toolchains.
  • Energy and benchmarking: Efficiency should be reported with standardized, power-inclusive benchmarks using delivered rather than nameplate performance.Peak TOPS-per-watt does not predict the energy of a real serving workload.
  • Energy and benchmarking: Reducing data movement through low precision, sparsity, and compute-near-data aligns sustainability goals with capability improvements.These architectural and algorithmic levers are identified as principal sustainability levers rather than secondary optimizations.
  • Hardware–algorithm co-design: Hardware–algorithm co-design should jointly advance quantization, sparsity, attention redesign, speculative decoding, and memory-aware hardware.A practical mixture-of-experts design for mobile power and memory budgets remains unsolved.
  • Heterogeneous systems: Future systems are likely to be heterogeneous and disaggregated, assigning compute-bound work to GPUs or NPUs and memory-bound attention to processing-in-memory.The review also identifies separate prefill and decode device pools and pooled memory over CXL and optical fabrics.
  • Selection guidance: The practical selection guide frames accelerator choice around workload characteristics rather than a universal platform.The review closes its unresolved-problems discussion with a heuristic guide in Figure 11.
  • Access and ecosystem: Accelerator access remains concentrated among a few vendors and well-resourced organizations, with cost and supply shaping participation.Open models, efficient inference, and more open hardware and toolchains may broaden participation, but the gap remains.

13 Conclusion

The review surveys accelerator architectures for LLM training and serving and identifies memory, rather than arithmetic, as the defining constraint. It concludes that heterogeneous hardware–algorithm co-design, not displacement by one architecture, is the credible path forward.

  • Scope: The review spans GPUs, custom ASICs, FPGAs, processing-in-memory, near-memory, neuromorphic, and photonic accelerators across cloud and edge deployment.Its organizing tension is LLM growth against the physical and economic limits of training and serving hardware.
  • Central finding: Memory, not arithmetic, is the defining constraint on LLM acceleration.Serving decode is bandwidth-bound, key-value cache size can rival model weights, and data movement consumes far more energy than computation.
  • Central finding: Memory-centric responses include high-bandwidth-memory scaling, processing-in-memory as a heterogeneous complement, and pooled memory over CXL and optical fabrics.These responses address the need to move less data rather than merely compute faster.
  • Platform roles: GPUs remain the flexible default and training workhorse, while ASICs win at scale for stable high-volume workloads and FPGAs serve latency-sensitive streaming niches.Processing-in-memory is positioned as a complementary architecture rather than a universal replacement.
  • Future direction: The main forward constraints are the memory wall and software gap, with CUDA maturity and portable compiler infrastructure remaining decisive.Alternatives are gated less by raw capability than by compiler and toolchain immaturity.
  • Conclusion: LLM acceleration is converging on a heterogeneous toolkit matched to workload diversity, with the memory system treated as the computer.Progress depends on jointly optimizing the model, its numerics, and the memory system.
Loading 2608.28048v1…