Source-linked AI summary
LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, Dongsoo Lee
TL;DR
Large language model generation faces memory constraints, and weight-only quantization kernels often require dequantization. LUT-GEMM performs matrix multiplication directly on quantized weights with full-precision activations; for 3-bit OPT-175B on one GPU, it reports 2.1× lower token-generation latency than OPTQ.
Problem
Large models strain GPU memory during generation, while weight-only quantization kernels reduce memory movement but incur dequantization overhead.
Source: §1 · pp. 1–2Method
Results
Takeaways & Limitations
LUT-GEMM supports uniform and non-uniform weight quantization, with reported latency and accuracy trade-offs across BCQ configurations.
Source: §1 · p. 2The method focuses primarily on single-batch inference, and its performance gains diminish as batch size increases because shared-memory bandwidth between cores and LUTs is constrained.
Source: §6 · p. 10
Abstract
from arXiv · showhide
Recent advances in self-supervised learning and the Transformer architecture have significantly improved natural language processing (NLP), achieving remarkably low perplexity. However, the growing size of NLP models introduces a memory wall problem during the generation phase. To mitigate this issue, recent efforts have focused on quantizing model weights to sub-4-bit precision while preserving full precision for activations, resulting in practical speed-ups during inference on a single GPU. However, these improvements primarily stem from reduced memory movement, which necessitates a resource-intensive dequantization process rather than actual computational reduction. In this paper, we introduce LUT-GEMM, an efficient kernel for quantized matrix multiplication, which not only eliminates the resource-intensive dequantization process but also reduces computational costs compared to previous kernels for weight-only quantization. Furthermore, we proposed group-wise quantization to offer a flexible trade-off between compression ratio and accuracy. The impact of LUT-GEMM is facilitated by implementing high compression ratios through low-bit quantization and efficient LUT-based operations. We show experimentally that when applied to the OPT-175B model with 3-bit quantization, LUT-GEMM substantially accelerates token generation latency, achieving a remarkable 2.1$\times$ improvement on a single GPU when compared to OPTQ, which relies on the costly dequantization process.
1 INTRODUCTION
Large language models’ growing size strains GPU memory, while distributing inference across GPUs adds communication overhead. LUT-GEMM targets weight-only quantized matrix multiplication without dequantization, and the paper reports faster OPT-175B token generation than OPTQ.
1 INTRODUCTION
- Billions-parameter models exceed single-GPU memory capacity, while model-parallel communication makes performance gains sub-linear as GPUs are added.Model parallelism distributes computation across GPUs but incurs inter-GPU communication overhead.Source: §1 · pp. 1–2
- Weight-only quantization offers higher compression at a target accuracy than quantizing weights and activations, but existing kernels dynamically dequantize weights before multiplication.Earlier weight-only approaches preserve full-precision activations but rely on conversion to full precision during computation.Source: §1 · p. 2
- LUT-GEMM uses BCQ to support uniform and non-uniform weight quantization, directly multiplying quantized weights with full-precision activations without on-the-fly dequantization.The paper says BCQ can represent both quantization types and describes LUT-GEMM as supporting a range of weight-only schemes.Source: §1 · p. 2
- For OPT-175B weights served by a single GPU, LUT-GEMM accelerates token-generation latency by 2.1× compared with OPTQ.Source: §1 · p. 3
2 BACKGROUND
Large-model inference is constrained by matrix-multiplication time and GPU memory bandwidth. The background reviews activation and weight quantization trade-offs, then introduces BCQ and its extension for uniform and group-wise quantization.
2.1 GPU-ACCELERATED GENERATIVE LMS
- Matrix multiplications account for at least 75% of processing time across various LM sizes and input lengths, while GPU performance depends on arithmetic intensity and memory reuse.Large models may also require multiple GPUs because of single-GPU memory limits, adding communication latency.Source: §2.1 · p. 3
2.2 QUANTIZATION METHODS AND LIMITATIONS
- Quantization is widely studied to improve latency and throughput; uniform INT8 is well-studied and actively applied in LLMs.The background also identifies pruning, distillation, and low-rank approximation as other approaches to improving deep-network serviceability.Source: §2.2 · p. 3
- INT8 quantization reduces latency and memory use, but generation is memory-bound and activation quantization can lose accuracy or yield only marginal gains.The text notes outlier approximation with offline activation scaling and contrasts memory-bound generation with compute-bound summarization.Source: §2.2 · p. 3
- Weight-only W4A16 compresses weights without quantizing activations, but OPTQ and AWQ dynamically dequantize weights to FP16 before matrix multiplication.This approach reduces memory movement during generation despite the dequantization overhead.Source: §2.2 · p. 4
2.3 BINARY-CODING QUANTIZATION
- BCQ represents weights using binary vectors and scaling factors; the paper extends it to cover uniform and group-wise quantization while retaining full-precision activations.The extension broadens BCQ’s applicability to different quantization methods.Source: §2.3 · p. 4
3 DESIGN METHODOLOGY OF LUT-GEMM
LUT-GEMM uses lookup tables to replace repeated arithmetic for quantized weights while retaining full-precision activations, and extends BCQ to support different quantization formats. Its GPU design and group-wise quantization expose trade-offs among latency, accuracy, and compression ratio.
3 DESIGN METHODOLOGY OF LUT-GEMM
- LUT-GEMM applies lookup-table computation directly to BCQ weights, avoiding dequantization while offering configurable trade-offs between compression ratio and quantization error.The method uses lookup tables to reduce redundant calculations and extends BCQ with a bias term.Source: §3 · p. 4
3.1 LUT BASED QUANTIZED MATRIX MULTIPLICATION
- LUT-GEMM precomputes full-precision activation combinations for binary patterns, then retrieves partial dot products from a table to avoid repeated computations.The partial products are summed and multiplied by scaling factors; the LUT becomes more useful as repeated patterns increase with matrix row size.Source: §3.1 · pp. 4–5
3.2 LUT BASED IMPLEMENTATION ON GPU
- GPU thread configurations balance parallel LUT access, shared scaling factors, and LUT utilization against synchronization overhead, so they are optimized empirically.Each thread block receives l LUTs and a B submatrix of size th × tw, with tw = l × µ.Source: §3.2 · p. 5, Figure 2 · p. 6
3.3 REPRESENTATIONAL CAPABILITY OF LUT-GEMM
- A bias term extends BCQ to represent both uniform and non-uniform quantization, using one scaling factor for uniform formats and q distinct factors for non-uniform formats.The bias also allows asymmetric quantization centered around z.Source: §3.3 · pp. 5–6
3.4 LATENCY-ACCURACY TRADE-OFF FOR IMPROVED APPLICABILITY
- Group-wise quantization with group sizes such as 128 and 256 can preserve fast LUT-GEMM while substantially improving accuracy.For single-batch operations, memory footprint explains latency: when g ≫ 16, space is approximately O(m · n · q), and similar footprints imply similar latency across (q, g) pairs.Source: §3.4 · pp. 6–7, Figure 4 · p. 7
4 EXPERIMENTAL RESULTS
Experiments progress from individual kernel comparisons through tensor-parallel profiling to end-to-end inference. They assess latency alongside perplexity, GPU utilization, and energy use.
4.1 KERNEL EVALUATION
- 2.6× less computation and the lowest latency among the compared kernels were reported for LUT-GEMM on OPT-175B’s first FFN layer.Unlike OPTQ and AWQ, LUT-GEMM accepts quantized weights directly, avoiding dequantization before GEMM.Source: §4.1 · p. 7, Table 1 · p. 8
- Tensor parallelism with cuBLAS reduced GPU and bandwidth utilization as communication latency rose, while LUT-GEMM retained high utilization on one GPU and saved energy.The authors attribute temporary GPU idling to synchronization and report that tensor-parallel speed-up is smaller than the number of GPUs used.Source: §4.1 · pp. 7–8, Table 2 · p. 8
4.2 END-TO-END LATENCY
- Smaller quantization groups reduced perplexity at a marginal latency cost, while adding GPUs did not significantly reduce end-to-end latency because of communication overhead.With LUT-GEMM, faster matrix multiplication makes relative GPU-to-GPU communication overhead more pronounced.Source: §4.2 · p. 8
- 2.41× speed-up for LLaMA-30B on one GPU and 2.04× for LLaMA-66B on two GPUs were achieved with 3-bit quantization and LUT-GEMM at g=128.The end-to-end evaluation also considers perplexity and latency across model and GPU configurations.Source: §4.2 · p. 8, Table 3 · p. 9
5 ACCELERATING QUANTIZED OPT-175B
This section evaluates LUT-GEMM with quantized OPT-175B weights. It reports lower token-generation latency than OPTQ and comparable latency while reducing the GPU requirement from eight to one.
5 ACCELERATING QUANTIZED OPT-175B
- 2.1× lower token-generation latency than OPTQ was reported for identical 3-bit, weight-only, row-wise quantization of OPT-175B.The comparison uses the FasterTransformer framework; LUT-GEMM directly accepts quantized weights, eliminating dequantization.Source: §5 · p. 9, Table 4 · p. 9
- BCQ quantization let LUT-GEMM run OPT-175B on one GPU instead of the eight required for FP16, while maintaining comparable overall latency.The latency comparison is for token generation on an A100 80GB.Source: §5 · p. 9, Table 4 · p. 9
- With OPTQ quantization, lowering q reduced LUT-GEMM latency, while an excessively small group size could marginally increase latency.Combining LUT-GEMM with OPTQ reduced the GPU requirement from eight to one at comparable latency, with an acceptable increase in perplexity.Source: §5 · p. 9
6 SUMMARY AND LIMITATIONS
The paper presents LUT-GEMM as a kernel that works directly on quantized weights and reports a speed-up over GPTQ. It also identifies a batch-size boundary for its performance gains.
6 SUMMARY AND LIMITATIONS
- LUT-GEMM processes quantized weights directly without dequantization and, using extended BCQ, reports a 2.1× speed-up over GPTQ.The method’s stated focus is efficient matrix multiplication for quantized models.Source: §6 · p. 9
- Table 5 covers OPT-175B perplexity under OPTQ quantization and LUT-GEMM per-token latency across q and g settings; its perplexity values come from the OPTQ reference.A dash for g denotes row-wise quantization.Source: Table 5 · p. 10, §6 · p. 10
- The method primarily targets single-batch inference, and its performance gains diminish as batch size increases because shared-memory bandwidth between the core and LUT is constrained.The authors anticipate that this constraint may become less significant with advanced hardware.Source: §6 · p. 10
7 REPRODUCIBILITY STATEMENT
The supplementary materials provide code and instructions for reproducing the Kernel Evaluation experiments.
7 REPRODUCIBILITY STATEMENT
- The supplied reproduction materials include the CUDA kernel and a concise benchmarking script for single matrix-vector products.These materials are specified for the Kernel Evaluation experiments.Source: §7 · p. 10
- The stated reproduction package covers the paper’s Kernel Evaluation experiments.The supplementary materials describe the CUDA kernel, benchmarking script, and README as components of that package.Source: §7 · p. 10
- A README provides example commands and instructions for running all scripts.Source: §7 · p. 10
A LLM INFERENCE LATENCY BREAKDOWN
The latency breakdown examines OPT models from 6.7B to 175B parameters on an A100 GPU. Matrix multiplications account for at least 75% of processing time, while larger models also incur GPU communication overhead.
A LLM INFERENCE LATENCY BREAKDOWN
- Matrix multiplications account for at least 75% of processing time across OPT model sizes and input token lengths.The analysis covers OPT models from 6.7B to 175B parameters.Source: A LLM INFERENCE LATENCY BREAKDOWN · p. 15
- Larger models show GPU communication overhead because a single GPU's memory capacity is limited, necessitating communication between GPUs.The latency breakdown was conducted on an A100 80GB GPU using FasterTransformer.Source: A LLM INFERENCE LATENCY BREAKDOWN · p. 15
- The latency analysis covers OPT models ranging from 6.7B to 175B parameters and varies the number of input tokens.The models were evaluated on an A100 80GB GPU with Nvidia's FasterTransformer inference framework.Source: A LLM INFERENCE LATENCY BREAKDOWN · p. 15
B DETAILED IMPLEMENTATION
The implementation builds lookup tables from partial inputs, shares them among threads, and combines partial outputs. Uniform quantized weights are converted into BCQ through a two-step construction of binary weights and per-bit scaling factors.
B DETAILED IMPLEMENTATION
- Threads in each thread block precompute LUTs from partial x values, share them to reduce global-memory accesses, and process submatrix rows in parallel.After retrieving and summing LUT values, threads fetch scaling factors once each and multiply them to produce partial outputs.Source: B DETAILED IMPLEMENTATION · p. 15
- LUTs reside in GPU shared memory, where their high bandwidth supports fast matrix computations by replacing multiple FLOPs with one LUT access.The passage states that 1KB suffices for LUTs per 8 hidden dimensions and gives A100 shared-memory capacity as 20MB.Source: B DETAILED IMPLEMENTATION · p. 15
C CONVERSION OF UNIFORM QUANTIZATION INTO BCQ
- Uniform quantization is converted to BCQ in two steps: generate scaling factors and integer weights, then transform the integer weights into binary weights.For bit position i, α_i=2^(i-1)s and b_i=2b̂_i−1, where s and b̂_i are the uniform-quantization scale and bit value.Source: C CONVERSION OF UNIFORM QUANTIZATION… · pp. 15–16
D IMPACT ON COMPRESSION RATIO
Group-wise BCQ uses quantization bits and group size to trade quantization error against scaling-factor memory, expanding the available compression choices. Experiments show that optimal configurations vary by model, so both parameters should be explored together.
D IMPACT ON COMPRESSION RATIO
- Smaller group size g can lower quantization error but increases scaling-factor memory, while q and g jointly determine the achievable compression ratio.Group-wise BCQ provides a wider search space than conventional row-wise quantization for meeting a target compression ratio.Source: D IMPACT ON COMPRESSION RATIO · p. 16
- A smaller q can still yield a larger memory footprint when g is small because more scaling factors are used.The illustrated example quantizes a 4 × 8 matrix and assumes 16-bit scaling factors.Source: D IMPACT ON COMPRESSION RATIO · p. 16, Figure 6 · p. 16
E EXPLORATION OF COMPRESSION RATIO
- Group-wise BCQ offers new optimal configurations compared with row-wise quantization, and q and g must be explored together for a target compression ratio or accuracy degradation.For OPT-13B and OPT-30B, small g is critical for low accuracy degradation, while latency is not heavily affected; q and g effects differ by model.Source: E EXPLORATION OF COMPRESSION RATIO · p. 17, Figure 7 · p. 17
F ADDITIONAL EXPERIMENTAL RESULTS
The appendix presents additional experimental results on perplexity, latency, and accuracy across OPT and LLaMA models and an MNLI evaluation. Some end-to-end latency values are omitted because of out-of-memory constraints.
F ADDITIONAL EXPERIMENTAL RESULTS
- Table 6 compares perplexity and end-to-end latency per token for OPT family models.The passage notes that reported PPL and accuracy numbers are extracted from the OPTQ reference.Source: Table 6 · p. 17, F ADDITIONAL EXPERIMENTAL RESULTS · p. 19
- Table 7 compares perplexity and end-to-end latency per token for LLaMA family models.Source: Table 7 · p. 18
- Table 8 reports accuracy for 2-bit AlphaTuning on MNLI, while Table 9 covers OPT-175B accuracy with OPTQ and corresponding LUT-GEMM latency.FP16 latency for OPT-66B and OPT-175B, and 4-bit OPT-175B latency, are omitted due to OOM constraints on a single A100 80GB GPU.Source: Table 8 · p. 18, Figure 8 · p. 18, Table 9 · p. 19