PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models

Cong Guo, Feng Cheng, Zhixu Du, James Kiessling, Jonathan Ku, Shiyu Li, Ziru Li, Mingyuan Ma, Tergel Molom-Ochir, Benjamin Morris, Haoxuan Shan, Jingwei Sun, Yitu Wang, Chiyue Wei, Xueying Wu, Yuhao Wu, Hao Frank Yang, Jingyang Zhang, Junyao Zhang, Qilin Zheng, Guanglei Zhou, Hai, Li, Yiran Chen

arXiv:2410.07265v1cs.ARcs.AIcs.LGcs.SE

TL;DR

LLM training and deployment bring substantial computational and memory demands, motivating specialized system-level optimization. This survey synthesizes algorithm, hardware, and system co-design research and examines its trade-offs, concluding that efficient LLM-centric systems require specialized solutions and further work on performance and sustainability.

  • Problem

    LLMs require substantial memory and computation for training and inference, creating a need to address their system-level resource demands.

    Source: §I · p. 1

  • Method

    The survey analyzes LLM challenges and synthesizes research on algorithm optimization, hardware architecture, and system-level innovations.

    Source: §I · pp. 1–2

  • Results

    The survey identifies specialized hardware and software solutions and trade-offs as central to understanding efficient LLM-centric computing systems.

    Source: §V · p. 13

  • Takeaways & Limitations

    The survey concludes that future research should advance methodologies and strategies to improve LLM performance, sustainability, efficiency, and practicality.

    Source: §V · p. 13

    Efficient LLM training remains a nascent field, with practical challenges and deployment difficulties limiting research, particularly in accelerator design.

    Source: C. Accelerators for Training · p. 6

Abstract

from arXiv · show

The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.

I. INTRODUCTION

LLMs offer expanding capabilities but impose substantial training and inference demands. The survey examines LLM-specific algorithm, hardware, and system design trade-offs, while the preliminaries situate LLMs within the transformer architecture.

I. INTRODUCTION

  • LLM training requires much more memory than CNN training, while autoregressive inference and growing KV caches create memory-management demands that increase with sequence length.The survey gives GPT-3's 175 billion parameters as requiring around 350GB of GPU memory for parameters alone, compared with about 100MB of weights for ResNet-50.Source: §I · p. 1
  • Unlike CNNs, LLM architectural consistency creates opportunities for custom operators tailored to particular architectures or hyperparameters.The survey’s roadmap covers LLM preliminaries, training practices, inference co-design, and its concluding contributions.Source: §I · pp. 1–2
  • The survey analyzes LLM challenges and their impact on hardware and algorithm research through algorithm optimization, hardware architecture, and system-level design trade-offs.It aims to synthesize research and identify trends to inform future LLM-centric computing systems.Source: §I · pp. 1–2

II. PRELIMINARIES

  • Transformers underpin modern LLMs, and their attention mechanism weighs the importance of different words in a sentence.The preliminaries introduce the transformer architecture as a response to limitations of earlier recurrent and convolutional models.Source: §II · p. 2

A. Transformer Architecture

The transformer projects input embeddings into queries, keys, and values, then uses self-attention and feed-forward layers to process representations. Residual connections and layer normalization wrap its sub-layers.

A. Transformer Architecture

  • Transformer models use an encoder-decoder structure, while many GPT-style LLMs use only the decoder.The architecture’s multi-head self-attention mechanism weighs the importance of different words in a sentence.Source: A. Transformer Architecture · p. 2
  • Input embeddings are linearly projected through learned weight matrices into queries, keys, and values for each attention head.The projections use X and the learned matrices WQ, WK, and WV.Source: Linear Projection · p. 2
  • Self-attention uses queries and keys to calculate scaled dot-product scores, applies SoftMax to obtain a probability distribution, and uses values to form the output.The key-vector dimension dk supplies the scaling factor for stable gradients.Source: Self-Attention Mechanism · p. 2
  • Multi-head attention runs self-attention operations in parallel with different parameter sets and concatenates their outputs to capture relationships across representation subspaces.The concatenated outputs are multiplied by WO.Source: Multi-Head Attention · p. 2
  • After attention, a feed-forward network applies linear transformations with nonlinear activations; each position is processed independently.Its learned parameters are W1, W2, b1, and b2, and σ is the activation function.Source: Feed-Forward Networks (FFN) · p. 2
  • Residual connections wrap each attention and feed-forward sub-layer, and layer normalization follows it to stabilize the input distribution and support faster training convergence.The cited passage also describes residual connections as helping gradients flow through deeper networks.Source: Residual Connections and Layer… · p. 2

B. Scope of the Survey on Large Language Models

The survey classifies transformer language models into encoder-decoder, encoder-only, and decoder-only types, while noting that “large language model” lacks a precise scope. Its focus is GPT-style models at least as large as GPT-2.

B. Scope of the Survey on Large Language Models

  • The survey groups transformer language models into encoder-decoder, encoder-only, and decoder-only types, associating the first two with BERT-style models and the last with GPT-style models.The classification follows prior categorizations, with the BERT-style and GPT-style labels linked to [15] and, respectively.Source: B. Scope of the Survey on Large… · p. 2
  • “Large language model” has no precise definition or scope, and prior work differs on whether BERT-style models count as large.Yang et al. consider BERT large while focusing predominantly on GPT-style models; Zhao et al. [16] call BERT-style models small-scale.Source: B. Scope of the Survey on Large… · p. 2
  • The survey focuses on GPT-style models with at least GPT-2’s 1.5 billion parameters, citing their popularity shift, scaling laws, and autoregressive mechanism as reasons.The passage says GPT-style models have dominated development since GPT-3’s introduction and notes their few-shot and zero-shot performance alongside hardware challenges.Source: B. Scope of the Survey on Large… · p. 3

III. TRAINING

LLM training is resource-intensive, especially pretraining, and requires memory, bandwidth, and performance optimizations. The survey covers parallelism, memory-management techniques, and training frameworks as parts of a broader hardware-software response.

III. TRAINING

  • Pretraining is costly in time, compute, and energy, and this survey uses “training” to mean the pretraining step rather than fine-tuning.Reported examples include over 196,000 TPUv3 core hours for a 600B model and over 1,720,000 GPU hours for Llama2-70B pretraining.Source: §III · p. 3
  • Training must store parameters, gradients, optimizer states, and activations, while scaling models and distributed or offloaded execution raise data-movement and bandwidth demands.The survey notes that improving performance and reducing energy consumption require attention to parallelism, data movement, and energy-efficient hardware.Source: §III · p. 3
  • Training solutions span infrastructure, hardware, and algorithms, with the survey organizing its discussion around system, algorithm, and accelerator levels.The framework-and-system discussion includes parallelism, offloading, rematerialization, LoRA, and training frameworks.Source: §III · p. 3, A. Framework and System · p. 3

A. Framework and System

  • Data parallelism duplicates the setup across data portions, while model parallelism splits work across GPUs through pipeline or tensor parallelism; FSDP is another training approach.Pipeline parallelism assigns layers to GPUs, whereas tensor parallelism divides tensors into chunks.Source: A. Framework and System · p. 3
  • ZeRO partitions optimizer states, gradients, and parameters to reduce redundant GPU data, while later variants add CPU or NVMe offloading and communication-efficiency measures.Offloading can enable larger models but increases GPU-CPU communication overhead and can slow training.Source: A. Framework and System · p. 4
  • Rematerialization stores selected activations and recomputes omitted ones during backpropagation, trading additional computation for lower memory use.GMLake addresses fragmentation from memory-reduction techniques by stitching non-contiguous blocks; specialized LLM training frameworks integrate optimization techniques.Source: A. Framework and System · p. 4

B. Algorithm and System Co-design

The section surveys parameter-efficient adaptation and related algorithmic approaches that reduce the burden of adapting or augmenting pretrained language models. It covers low-rank and selective parameter updates, prompting, and retrieval-augmented generation.

B. Algorithm and System Co-design

  • LoRA adapts pretrained weights through low-rank decomposition, updating smaller matrices instead of all parameters to reduce fine-tuning time and memory while retaining competitive performance.It applies a low-rank decomposition to weight matrices to approximate the necessary changes.Source: B. Algorithm and System Co-design · p. 4
  • QLoRA fine-tunes a 65B-parameter LLaMA on one 48GB GPU with performance similar to a 16-bit full-finetuned model.It uses 4-bit NormalFloat quantization, Double Quantization, and Paged Optimizers; LQ-LoRA extends quantization below 3 bits.Source: B. Algorithm and System Co-design · p. 4
  • Prompt-based learning steers pretrained language models with cues or examples; continuous prompt tuning optimizes input-embedding vectors while keeping model parameters unchanged.The passage notes that manually crafted prompts can underperform fine-tuned models.Source: B. Algorithm and System Co-design · pp. 4–5
  • Retrieval-augmented generation combines language generation with external knowledge retrieval, commonly using approximate nearest-neighbor search for high-dimensional vector databases.The section describes HNSW, DiskANN, and Faiss as retrieval approaches.Source: B. Algorithm and System Co-design · p. 5
  • Other PEFT approaches localize adaptation by selecting parameters or masks, including task-specific sparse masks, a universal parameter set, and Fisher-information-based masks.Diff pruning, PaFi, and FISH Mask exemplify these strategies.Source: B. Algorithm and System Co-design · p. 5

C. Accelerators for Training

The survey describes memory-focused accelerator designs for LLM training, then turns to inference deployment and system-level optimization. It distinguishes offline latency reduction from online serving strategies aimed at throughput and memory management.

C. Accelerators for Training

  • LLM training accelerators emphasize memory-centric optimization, especially memory compression, because of the models’ exceptionally large memory footprints.The survey describes GPUs, TPUs, and ASIC designs addressing these demands.Source: C. Accelerators for Training · p. 5
  • GPU Transformer Engine uses mixed-precision techniques, while TPUs use systolic arrays and high memory bandwidth to support LLM training and inference.The Transformer Engine combines FP16 and FP32 computations; TPUs accelerate matrix multiplication.Source: C. Accelerators for Training · p. 5
  • Smart-Infinity extends host memory and storage for LLM training with near-storage parameter updates, reduced storage traffic, overlapped data transfers, and accelerator-assisted gradient compression.Earlier sparsity- and quantization-based studies mainly targeted smaller language models such as GNMT.Source: C. Accelerators for Training · pp. 5–6
  • Efficient LLM training remains nascent, with practical challenges and deployment difficulties limiting research, particularly in accelerator design.The survey shifts to inference optimization studies, which it describes as having lower complexity and broader applicability.Source: C. Accelerators for Training · p. 6

IV. INFERENCE

  • The largest pretrained models can impose computational and memory costs impractical for most consumer-grade devices, motivating work on accelerators, compression, and algorithmic advances.The passage gives LLaMa-70B as an example.Source: §IV · p. 6

A. LLM Inference System

  • Offline inference targets latency for one user’s initially submitted requests, whereas online serving handles asynchronous multi-user requests and targets throughput through memory management, batching, and scheduling.Deployment across multiple devices also makes system-level optimization a research focus.Source: A. LLM Inference System · p. 6

1) Inference Engine:

Inference-system optimizations target faster model execution and more responsive serving through operator fusion, memory management, batching, and scheduling. The survey also describes heterogeneous-device approaches and notes that their practical trade-offs need real-world performance data.

1) Inference Engine:

  • Kernel fusion combines operators or layers to reduce memory access and launch overhead while improving parallelism; attention and linear operations account for over 75% of inference duration [71].FlashAttention, [73] integrates attention computation into one operator and reports 3× training speed up on GPT-2.Source: 1) Inference Engine · p. 6
  • Inference frameworks fuse lightweight operations such as LayerNorm and SwiGLU, while specialized GEMV and FlatGEMM implementations target linear operations [75],.TensorRT-LLM also uses pattern matching to identify potential fusions across LLM architectures.Source: 1) Inference Engine · p. 6
  • PagedAttention segments the KV cache into blocks and reports 24× higher throughput than HuggingFace Transformers [34], while offloading methods use CPU, GPU, and disk resources,.ZeRO-Inference overlaps computation with weight fetching; FlexGen uses linear programming to optimize offloading.Source: 1) Inference Engine · p. 6

2) Online Serving:

  • Continuous batching adds new requests as earlier ones complete, while split-and-fuse methods balance work across processing stages to reduce response times and improve throughput –.The approaches include ORCA, Sarathi [83], Sarathi-Serve [84], and LightLLM.Source: 2) Online Serving · p. 6
  • Serving systems reduce KV-cache waste through predicted generation-length bounds, paged storage, and token-level allocation,,.LLM in a Flash [81] stores model parameters in flash memory and dynamically loads them into DRAM when model size exceeds DRAM capacity.Source: 2) Online Serving · p. 6
  • FastServe prioritizes requests by estimated completion time, VTC adjusts resource allocation for fairness, and distributed approaches address preemptible GPUs or separate prefilling from decoding [87]–[91].First-come-first-served scheduling can cause head-of-line blocking,,.Source: 2) Online Serving · p. 7
  • On limited-resource devices, inference may transfer parameters and intermediate results between storage and DRAM or between host memory and GPU; reducing transfer costs is a key concern [81], [92]–[95].Leveraging predictable activation sparsity can reduce the parameters needed for computation and thereby reduce transfer and storage costs [81], [92], [95]–[97].Source: 2) Online Serving · p. 7
  • The approaches involve trade-offs among computational efficiency, memory usage, and response times, and real-world performance data would help quantify them across deployment scenarios.The passage identifies latency constraints and hardware limitations as relevant selection considerations.Source: 2) Online Serving · p. 7

B. Algorithm for Efficient LLM

The section surveys algorithmic approaches for faster LLM inference, including Mixture-of-Experts (MoE), efficient attention, speculative decoding, and state-space models. It describes MoE’s expert routing and reviews work on scaling, stability, load balancing, and throughput.

B. Algorithm for Efficient LLM

  • The survey identifies MoE, efficient attention, speculative decoding, and structured state-space models as approaches targeting faster feed-forward computation, attention, autoregression, or long-sequence efficiency.It presents SSMs as an alternative to transformers for improving computational efficiency over long sequences.Source: B. Algorithm for Efficient LLM · p. 7

1) MoE:

  • An MoE layer routes each input entity to selected expert models, and scaling its parameter count can incur little computational overhead, supporting its use for fast inference.The survey names GPT-4 [102] and Mixtral [103] as LLMs employing MoE.Source: 1) MoE · p. 7
  • MoE research spans large-scale models, including a 137B-parameter LSTM [104], a 1.6T-parameter Switch Transformer, and GShard’s expansion by over 600B parameters [18].Clark et al. studied routing-based language-model scaling and derived an Effective Parameter Count.Source: 1) MoE · pp. 7–8
  • Further MoE work addresses training stability and balanced routing, activates only some weights per forward pass, and develops throughput and expert-caching strategies [107], [112]–.Other surveyed directions include MoE for vision and reinforcement learning and Fusion of Experts [108]–[111],.Source: 1) MoE · p. 8

2) Efficient Attention:

Transformer attention is a time- and memory-intensive bottleneck, motivating methods that reduce key/value traffic, approximate attention, or replace attention-based sequence processing. Speculative decoding and state-space models offer additional approaches to inference efficiency.

2) Efficient Attention:

  • Attention is a transformer computation bottleneck in both time and memory, with efficient methods grouping keys and values or approximating attention scores.These are the two main research directions identified for efficient attention.Source: 2) Efficient Attention · p. 8
  • MQA shares averaged keys and values across heads to reduce memory bandwidth and speed decoding, while GQA relaxes sharing to address potential quality degradation.GQA-1 reduces to MQA, whereas GQA-H corresponds to multi-head attention with H heads.Source: 2) Efficient Attention · p. 8
  • Attention approximation reduces attention-matrix computation from O(n^2) to O(n), using kernel-based feature maps or low-rank compression.Kernel methods include random-feature and sketching approaches; low-rank methods compress Q and K, and FLuRKA combines both approaches.Source: 2) Efficient Attention · p. 8
  • Speculative decoding uses a small model to draft tokens for the LLM to accept or reject, reducing memory I/O and latency while potentially increasing FLOPs.Research seeks higher acceptance through parallel draft candidates or improved alignment between the small and large models.Source: 2) Efficient Attention · p. 8
  • State-space models compress sequence inputs into a state, giving linear computation and memory because each next token interacts with the state rather than all prior tokens.Research improves transition-matrix parameterization and computation; Mamba2 reports 2–8× speedup over Mamba, while MambaFormer and Jamba combine SSM and transformer components.Source: 2) Efficient Attention · pp. 8–9

3) Multi-modal LLMs.:

Multimodal LLMs align vision embeddings with text embeddings, but large numbers of vision tokens increase computational cost. The survey also reviews model compression, including quantization methods and hardware, and notes limits to current token-reduction and low-bit approaches.

3) Multi-modal LLMs.:

  • Vision encoders and projectors align image or video embeddings with a text-only LLM, but hundreds or thousands of vision tokens add computational burden.Perceiver learned pooling and PruMerge token selection and merging reduce vision-token costs, though systematic exploration remains limited.Source: 3) Multi-modal LLMs · p. 9

C. Compression Methods and Accelerators

  • Model compression targets the computational and resource demands of LLM deployment through quantization, pruning, knowledge distillation, and low-rank factorization.The survey frames these as four main compression categories.Source: C. Compression Methods and Accelerators · p. 9

1) Quantization:

  • Quantization-aware training retrains models for quantization noise, whereas post-training quantization converts floating-point models to lower-bit representations without training data.The latter is described as particularly suitable for large-scale language models.Source: 1) Quantization · p. 9
  • Quantization methods use scaling, protected weights, transformations, or adaptive rounding to support lower-bit LLM inference while managing outliers and quantization error.Examples include SmoothQuant, AWQ, QuaRot, and QuIP.Source: 1) Quantization · p. 9
  • Quantization accelerators and reduced-precision formats target efficient inference, but extremely low bit widths may add decoding overhead that limits proportional gains.The surveyed designs include bit-width-flexible processing elements, outlier-aware hardware, and 6-bit, 4-bit, and 8-bit formats; growing contexts may also require KV-cache quantization.Source: 1) Quantization · pp. 9–10

2) Sparsity:

LLM sparsification reduces computation, memory occupancy, or bandwidth through weight pruning and sparse attention. Its realized efficiency depends on the sparsity pattern and hardware support, with structured sparsity often favored in current practice.

2) Sparsity:

  • Weight pruning zeros less important parameters; structured pruning creates regular patterns suited to dense-workload accelerators, while unstructured pruning can offer higher compression but needs hardware support to use sparsity efficiently.SparseGPT reaches up to 50% weight sparsity in GPT models, but may not efficiently reduce computation on accelerators optimized for dense workloads.Source: 2) Sparsity · p. 10
  • Sparse attention limits which tokens attend to one another, reducing attention complexity and memory access through static, input-agnostic patterns or dynamic, input-dependent patterns.Dynamic methods detect activations at runtime; examples include SpAtten's token or head pruning and DOTA's finer-grained sparsity.Source: 2) Sparsity · pp. 10–11
  • Sparsity gains require hardware consideration: unstructured patterns need dedicated designs and add overhead, so current practice often favors structured sparsity for its regularity and balance of performance and hardware cost.Accelerators such as DS-STC and RM-STC target sparse matrix multiplication, while sparse NVPIM maps static sparse attention to processing-in-memory architecture.Source: 2) Sparsity · pp. 10–11

D. Accelerators for Inference

Inference accelerators combine hardware changes with algorithmic and architectural techniques to reduce data movement, exploit sparsity, or distribute computation. The surveyed designs also use PIM and heterogeneous systems to place attention and other computations on different resources.

D. Accelerators for Inference

  • Inference acceleration targets costly data movement and greater parallelism, with energy-efficient dedicated processors particularly relevant to power-constrained edge devices.The survey groups representative accelerator work around hardware and algorithmic approaches.Source: D. Accelerators for Inference · p. 11
  • TranCIM performs multiply-accumulate operations locally within SRAM arrays, avoiding costly intermediate-data transfers while activating bitlines in parallel.Its data arrangement also avoids transpose buffers for generated self-attention matrices, and different matrix multiplications can run in a pipeline.Source: D. Accelerators for Inference · pp. 11–12
  • Algorithmic acceleration dynamically exploits workload patterns or input complexity to skip computation, reconfigure data paths, and adjust local power and clock settings.The surveyed techniques report higher throughput and energy efficiency with marginal accuracy loss; PIVOT dynamically adjusts attention based on input complexity.Source: D. Accelerators for Inference · p. 12
  • Architectural designs support sparse attention through token-parallel dataflow and key/value reuse, or approximate and prune low attention scores with analog thresholding.SpecPIM explores resource allocation for the heterogeneous draft and target models used in speculative decoding.Source: D. Accelerators for Inference · p. 12
  • PIM and distributed systems divide inference work across specialized or multiple devices, including attention on PIM and other computations on xPUs.TransPIM avoids costly inter-layer data movement; DFX uses model parallelism across FPGAs, while CXL-PNM partitions workloads using model and data parallelism.Source: D. Accelerators for Inference · p. 12

E. Industry-led AI accelerators

Industry-led accelerators emphasize broad workload coverage and system-level co-design, while other work explores spiking and in-memory Transformer designs. The survey concludes that efficient, practical LLM systems require specialized hardware and software solutions.

E. Industry-led AI accelerators

  • Industry accelerators target varied AI workloads across training and inference, unlike academic designs often restricted to particular models.Examples include IBM's RaPiD, supporting precision down to 2-bit fixed point; NorthPole's architecture and SDK toolchain; Azure Maia 100's end-to-end system optimization; and Cerebras's wafer-sized architecture.Source: E. Industry-led AI accelerators · p. 12

F. Other optimizations

  • Spiking Transformer work reduces operations through spiking self-attention and random spike masking, while C-Transformer adds accumulation and output-spike speculation hardware.Spikingformer incorporates spike-driven residual learning; C-Transformer's HMAU accumulates spiking operations and its OSSU speculates output spikes.Source: F. Other optimizations · p. 13
  • In-memory Transformer designs span RRAM floating-point computation, analog content-addressable operations, attention-block reuse, and KV-matrix clipping.RIME performs floating-point operations in memory; RACE-IT handles non-matrix-vector operations in the analog domain; TReX reuses attention blocks to improve energy efficiency and area utilization while maintaining accuracy; ClipFormer clips KV matrices to address memristive-crossbar noise.Source: F. Other optimizations · p. 13

V. CONCLUSION

  • The survey calls for specialized hardware and software solutions and further work to improve LLM systems' efficiency, sustainability, and practicality.It frames these needs around trade-offs and design considerations for LLM-centric computing systems.Source: §V · p. 13
Loading 2410.07265v1…