PaperlayerView exact revision on arXiv

Source-linked AI summary

PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

Yixin Song, Zeyu Mi, Haotong Xie, Haibo Chen

arXiv:2312.12456v2cs.LGcs.OS

TL;DR

Consumer-grade GPUs cannot readily accommodate large LLMs, while existing offloading and sparse-inference approaches face memory, prediction, and data-transfer constraints. PowerInfer exploits activation locality by placing hot neurons on the GPU and cold neurons on the CPU, using adaptive predictors, neuron-aware operators, and offline placement. It achieves up to 11.69× faster inference than llama.cpp without compromising accuracy, while supporting models including OPT-175B on a single RTX 4090.

  • Problem

    Large LLMs exceed consumer-GPU memory, while existing offloading and activation-sparsity methods remain constrained by CPU-GPU transfers, predictor memory, and deployment limitations.

  • Method

    PowerInfer preloads frequently activated hot neurons on the GPU, computes input-dependent cold neurons on the CPU, and adds adaptive predictors, neuron-aware operators, and ILP-based placement.

  • Results

    Up to 11.69× faster inference than llama.cpp was achieved without compromising accuracy across supported LLMs, including OPT-175B, on a single NVIDIA RTX 4090.

  • Takeaways & Limitations

    PowerInfer enables fast local inference for models that exceed a consumer GPU’s capacity by exploiting activation locality across GPU and CPU resources.

  • Takeaways & Limitations

    The offline placement solver uses ILP, whose NP-completeness makes directly solving placement for models with hundreds of billions of parameters computationally challenging.

Abstract

from arXiv · show

This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key principle underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. This distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated across inputs, while the majority, cold neurons, vary based on specific inputs. PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. PowerInfer further integrates adaptive predictors and neuron-aware sparse operators, optimizing the efficiency of neuron activation and computational sparsity. The evaluation shows that PowerInfer significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU. For the OPT-30B model, PowerInfer achieves performance comparable to that of a high-end server-grade A100 GPU, reaching 82% of its token generation rate on a single consumer-grade RTX 4090 GPU.

1 Introduction

PowerInfer targets local LLM inference, where model memory demands and costly CPU-GPU transfers limit consumer-grade GPU deployments. It exploits activation locality by placing frequently activated neurons on the GPU and handling input-dependent neurons on the CPU, with adaptive predictors and neuron-aware operators.

  • Local LLM deployments prioritize low latency, privacy, customization, and reduced inference costs, unlike data-center deployments focused on high throughput.
  • Consumer GPUs struggle to hold LLMs because each autoregressive token requires access to the entire model.
  • 40GB of memory is required to load a 4-bit OPT-66B model, exceeding the NVIDIA RTX 4090’s capacity.
  • Existing layer-level offloading reduces GPU requirements but suffers high latency from slow PCIe transfers and limited CPU computation.
  • Over 80% of activations come from a small subset of consistently active hot neurons, while cold-neuron activations vary by input.
  • PowerInfer preloads hot and cold neurons onto GPU and CPU respectively, then uses online predictors to select activated neurons and reduce PCIe transfers.
  • Adaptive predictors reduce predictor size for sparse layers while maintaining accuracy, freeing GPU memory for LLM inference.
  • Neuron-aware sparse operators process individual neurons directly, avoiding whole-matrix operations and sparse-format conversions.

2 Background and Motivation

LLM inference activates only parts of Transformer computation, creating opportunities for sparse execution, but existing offloading approaches remain constrained by GPU capacity, CPU speed, and data movement. PowerInfer’s motivation follows from matching sparse activation patterns to GPU-CPU execution.

  • LLM Inference & Architecture: Each token-generation iteration runs the full autoregressive LLM model after prompt processing begins.
  • LLM Inference & Architecture: ReLU selectively retains positive FC1 values, determining which neurons contribute to the output passed into FC2.
  • LLM Inference & Architecture: Transformer layers combine self-attention with MLP blocks, whose FC1 and FC2 layers generate vectors through matrix multiplication.
  • LLM Inference & Architecture: Approximately 80% of OPT-30B neurons remain inactive during inference, while LLaMA2-13B and Yi-34B show 43% and 53% sparsity respectively.
  • LLM Inference & Architecture: DejaVu predicts activations online and processes only activated neurons, achieving over 6x speedup with at least 93% prediction accuracy.
  • Offloading-based LLM Serving: GPU-centric offloading transfers CPU-held weights during inference, causing over 99.5% of processing time to be spent moving weights from CPU to GPU.
  • Offloading-based LLM Serving: DejaVu faces consumer-GPU limitations because activated neurons must be transferred from CPU to GPU at runtime.
  • Offloading-based LLM Serving: Hybrid layer-level offloading reduces data transfer and reaches around 600ms latency, but remains slower than the 45ms latency of OPT-30B on A100.

3 Insights into Locality in LLM Inference

LLM neuron activations follow a power-law distribution: a small, frequently activated subset accounts for most activations, while the remainder depends on inputs. This locality extends across layers and tasks, and CPU execution can outperform GPU transfer for sparse, small-batch workloads.

  • 3.1 Insight-1: Power-law Activation: Figure 4 plots neuron proportion against cumulative activation frequency for single MLP blocks and the entire model.The curves compare OPT-30B, LLaMA2(ReGLU)-70B, and LLaMA2(SwiGLU)-70B.
  • 3.1 Insight-1: Power-law Activation: 26%, 43%, and 69% of neurons account for 80% of activations in single MLP blocks of OPT-30B, LLaMA2(ReGLU)-70B, and LLaMA2(SwiGLU)-70B, respectively.The remaining neurons are classified as cold-activated because their activation is input-dependent.
  • 3.1 Insight-1: Power-law Activation: Approximately 17%, 26%, and 75% of neurons account for 80% of activations across OPT-30B, LLaMA2(ReGLU)-70B, and LLaMA2(SwiGLU)-70B, respectively.In OPT-30B, the initial 24 layers have less than 1% active neurons and therefore few hot neurons.
  • 3.1 Insight-1: Power-law Activation: Over 90% of the top 20% most frequently activated neurons overlap across knowledge, truthfulness, and reasoning tasks.The paper interprets this consistency as evidence that power-law activation is an intrinsic model property rather than dataset-specific.
  • 3.2 Insight-2: Fast In-CPU Computation: For batch sizes under 32, direct CPU execution of CPU-resident neurons is faster than transferring their weights to an NVIDIA RTX 4090 and computing them there.The comparison uses OPT-30B and CPU AVX2 vector extensions for sparse MLP and attention computations.

4 PowerInfer Overview

PowerInfer is a single-consumer-GPU inference system that profiles neuron locality offline and coordinates GPU and CPU execution online. It places frequently activated neurons on the GPU, computes cold neurons on the CPU, predicts active neurons, and uses hardware-aware placement.

  • 4 PowerInfer Overview: PowerInfer preloads frequently activated hot-neuron weights on the GPU and keeps less active cold-neuron weights on the CPU.This neuron-aware offloading strategy uses both memories to support LLMs of varying sizes on a PC with one consumer-grade GPU.
  • 4 PowerInfer Overview: PowerInfer’s effectiveness correlates with activation sparsity: ReLU-family LLMs with over 90% sparse FFNs are ideal candidates, while models with around 50% sparsity gain less.The paper states that applicability may broaden as sparse or ReLU-family LLMs become more common.
  • 4.1 Architecture and Workflow: The offline component profiles activation sparsity, separates hot and cold neurons, and uses an integer-linear-programming solver to maximize GPU impact under hardware constraints.The profiler collects activation data from general datasets, while the solver considers activation frequencies, neuron impact, and hardware specifications.
  • 4.1 Architecture and Workflow: During online inference, predictors skip neurons predicted inactive, GPU executors process preloaded hot neurons, and CPU executors process cold neurons without transferring their weights to the GPU.The engine manages concurrent CPU-GPU computation and combines the resulting outputs on the GPU.
  • 4.1 Architecture and Workflow: Hot and cold neuron assignments are based on offline statistics, but hot labels may not match runtime activation for every input.For example, a neuron classified as hot can be predicted inactive for a particular request.

5 Neuron-aware Inference Engine

PowerInfer’s inference engine combines adaptive activation predictors with neuron-aware sparse operators and coordinated CPU-GPU execution. These designs reduce predictor memory, avoid unnecessary sparse-format conversions, and process only predicted active neurons.

  • 5.1 Adaptive Sparsity Predictors: Within each Transformer layer, predictors forecast active neurons so inference computes only the predicted active subset instead of the full neuron set.This prediction-based sparsity mechanism follows the same computational restriction used by DejaVu, while PowerInfer adapts predictor sizing for local deployments.
  • 5.1 Adaptive Sparsity Predictors: Adaptive predictors use smaller models for layers with higher activation sparsity and skewness, addressing the GPU-memory cost of fixed-size predictors.Predictor size is adjusted iteratively to preserve predictive accuracy while freeing memory for LLM parameters.
  • 5.1 Adaptive Sparsity Predictors: PowerInfer limits predictor parameters to 6% of total LLM parameters while keeping integrated-model perplexity within 0.1% of the baseline.Predictor hidden-layer size is iteratively tuned using WikiText-2 perplexity as the accuracy benchmark.
  • 5.3 Hybrid Execution: The engine assigns GPU and CPU executors to dependency-tagged operators in a global DAG and performs GPU-side merging after both units finish their neuron computations.Neuron tables preserve each segmented neuron’s original matrix position during multiplication.
  • 5.2 Neuron-aware Sparse Operators: Neuron-aware operators process activated neuron vectors directly on GPU and CPU, avoiding runtime dense-format conversion and whole-matrix operations.They first determine activation status, then compute only the corresponding row or column of the parameter matrix.
  • 5.2 Neuron-aware Sparse Operators: On GPUs, vector-vector neuron-aware computation is advantageous at small batch sizes because it avoids inactive-neuron work and sparse-matrix conversions.On CPUs, multiple cores process batches of activated neurons using vector extensions such as AVX2.

6 Neuron Placement Policy

PowerInfer formulates neuron placement as an integer linear program that maximizes GPU-assigned neuron impact while accounting for communication and memory constraints. Offline profiling and batching make this optimization practical for large models.

  • Profiling: PowerInfer profiles neuron activation on general datasets to estimate activation frequency and guide hot-neuron placement.The profiler monitors activations across model layers before placement decisions are made.
  • Impact metric: The neuron impact metric is based on each neuron’s profiled activation frequency and feeds the GPU allocation objective.
  • ILP formulation: The ILP assigns neurons to processing units while maximizing cumulative GPU impact and enforcing one-unit placement for each neuron.Binary variables indicate whether a neuron is placed on a processing unit.
  • Constraints: The placement solver minimizes communication overhead and respects GPU memory capacity, including a minimum GPU allocation required to offset transfer costs.Allocating too few neurons to a layer can negate the GPU’s computational benefit because of PCIe communication overhead.
  • Scalability: Because ILP is NP-complete, PowerInfer batches 64 similarly impactful neurons, reducing the problem from millions of neurons to tens of thousands and solving it in about 10 seconds.

7 Implementation

PowerInfer extends llama.cpp with GPU-CPU hybrid execution and neuron-aware operators, while its offline tools train activation predictors using general corpora. The implementation supports several major LLM families and parameter scales.

  • Online engine: PowerInfer extends llama.cpp with 4,200 lines of C++ and CUDA code for model distribution and GPU-CPU hybrid inference.The extensions include a modified model loader and 10 neuron-aware operators.
  • Offline component: Its offline component uses profilers and solvers, while activation predictors are trained on general corpora rather than downstream task datasets.Predictor training is a one-time task that can take several hours and can be accelerated with multiple GPUs.
  • Model support: PowerInfer supports mainstream LLM families with varying parameter sizes, including OPT, LLaMA2, and Falcon models.

8 Evaluation

PowerInfer is evaluated on high-end and low-end PCs across multiple LLM families, precisions, workloads, and batch sizes. It generally improves end-to-end generation performance over local baselines, while gains depend on hardware capacity and workload characteristics.

  • Experimental setup: The evaluation covers PC-High with an RTX 4090 and PC-Low with an RTX 2080Ti, using models from 13B to 175B parameters.Experiments use FP16 and INT4 parameters with FP32 intermediate activations.
  • Experimental setup: PowerInfer is compared with llama.cpp and SpecInfer using batch-one workloads sampled from Alpaca and ChatGPT prompts with variable input and output lengths.
  • FP16 performance: 8.32 tokens/s is PowerInfer’s average generation speed on PC-High, reaching 16.06 tokens/s and outperforming the compared systems.
  • FP16 performance: PowerInfer averages 4.71× and 5.97× speedups on PC-Low comparisons, peaking at 7.06× and 7.47×, with smaller gains limited by 11GB GPU memory.
  • Neuron load: PowerInfer raises the GPU share of activated-neuron computation from 20% to 70% on PC-High, but the share falls to 42% when model memory greatly exceeds GPU capacity.
  • Input length: For long input sequences, PowerInfer achieves 3.47× to 5.69× speedup by using dense GPU computation during prefill and sparse execution during generation.
  • Quantization and batching: With INT4 quantization, PowerInfer reaches 13.20 tokens/s on average and 29.08 tokens/s at peak on PC-High, with average speedup of 2.89× over llama.cpp.
  • Quantization and batching: For batch sizes below 32, PowerInfer achieves an average 6.08× performance improvement over llama.cpp, with the speedup decreasing as batch size increases.

8.3 Ablation Studies

The ablation studies show that PowerInfer’s predictors and neuron-aware operators, hybrid execution engine, and optimized partitioning policy progressively improve inference speed. Additional evaluations examine latency consistency, sparse-operator efficiency, predictor overhead, and the remaining gap to A100 performance.

  • Component Ablation: 2.60× and 7.80× speedups follow the introduction of the hybrid engine for Bamboo-7B and Falcon-40B.The engine processes neurons within the same layer on both GPU and CPU, increasing the GPU’s computational share.
  • Robustness: PowerInfer’s latency remains consistent across tasks, with P95 latency only 10% above average as sparsity varies from 80% to 86%.Tokens with lower sparsity introduce 10% more latency than average.
  • Neuron-aware Operator Performance: PowerInfer’s CPU sparse operator outperforms dense matrix multiplication below 10% sparsity, whereas traditional sparse operators require more than 87% sparsity.On GPUs, PowerInfer matches PIT while additionally supporting unified CPU-GPU execution.
  • Predictor Overhead: Predictor execution accounts for less than 10% of total inference time, while predictor parameters comprise 7.09% of model weights.Adaptive construction methods reduce predictor size and computational load.
  • Performance Comparison with A100: PowerInfer reduces the RTX 4090-to-A100 generation gap from 93% to 18% for OPT-30B and from 92% to 23% for Falcon-40B.For input length 64, the remaining gaps are 28% and 29%, mainly because CPU computation becomes a bottleneck.

8.4 SiLU-based LLM Performance

PowerInfer also accelerates SiLU-based LLMs, although its gains are smaller than for ReLU-based models. The evaluations additionally report negligible accuracy loss and show that a larger model can retain its accuracy while reaching smaller-model decoding speed.

  • SiLU-based LLM Performance: 1.47× to 1.7× speedups are achieved on SiLU-based LLMs.The gains are less pronounced than for ReLU-based counterparts but demonstrate effectiveness across activation functions.
  • LLM Accuracy: PowerInfer causes negligible inference-accuracy loss across OPT, Falcon, and LLaMA2 model families.The comparison covers models with and without differentiated activated and inactivated neurons across representative downstream tasks.
  • LLM Accuracy: Mispredicted neurons typically have minimal influence on layer outputs, helping explain the limited accuracy impact.The paper uses output cosine-similarity analysis on OPT-7B to quantify this effect.
  • Comparison with Smaller Language Models: Bamboo-7B-PowerInfer preserves the dense model’s accuracy while achieving 4B-level decoding speed.Bamboo-7B outperforms Qwen-1.5-4B across various tasks in the comparison.

9 Related Work

Related work addresses model weight sparsity, attention sparsity, speculative decoding, and general LLM serving, but these approaches differ from PowerInfer’s focus on intrinsic sparse activations for PC deployment.

  • LLM Weight Sparsity: Model pruning methods such as SparseGPT and Wanda reduce parameters by setting weights to zero, achieving 50% sparsity.These methods target weight sparsity rather than the intrinsic sparse activations exploited by PowerInfer.
  • LLM Weight Sparsity: SparTA and Flash-LLM optimize sparse tensor computation but are described as orthogonal to intrinsic LLM sparse activations.The paper notes accuracy-loss and wall-clock acceleration challenges for these approaches.
  • LLM Attention Sparsity: Attention-sparsity methods reduce computation through selective attention heads or KV-cache pruning and offloading.PowerInfer instead leverages sparse activation characteristics in MLP layers.
  • Speculative LLM Inference: Speculative inference uses a smaller model to pre-decode tokens and the main model to validate them, reducing decoding steps.The paper identifies integration with PowerInfer as a possible future speedup rather than part of its current focus.
  • LLM-Specific Serving Optimizations: Serving systems such as Orca and vLLM optimize scheduling or KV-cache storage but do not address PC deployment when the full model cannot fit in GPU memory.This boundary distinguishes them from PowerInfer’s target setting.

10 Conclusion

PowerInfer is an LLM inference system designed around locality in LLM inference. It combines adaptive predictors and neuron-aware operators to achieve substantial speedups without compromising accuracy.

  • Conclusion: PowerInfer achieves up to 11.69× faster LLM inference than systems such as llama.cpp without compromising accuracy.The system exploits LLM locality through adaptive predictors and neuron-aware operators for activation and computational sparsity.
Loading 2312.12456v2…