PaperlayerView exact revision on arXiv

Source-linked AI summary

Which Quantization Should I Use? A Unified Evaluation of llama.cpp Quantization on Llama-3.1-8B-Instruct

Uygar Kurt

arXiv:2601.14277v1cs.LG

TL;DR

Choosing among llama.cpp quantization formats is difficult because evaluations often emphasize perplexity, mix models, and offer limited unified downstream evidence. This study compares GGUF formats on Llama-3.1-8B-Instruct across tasks and deployment measures, finding that mid-bit quantization preserves capabilities surprisingly well and that 5-bit configurations offer a strong trade-off in this setting.

  • Problem

    Existing evaluations often focus on perplexity, combine heterogeneous models, and rarely report downstream performance in a unified way, leaving users without principled comparisons of llama.cpp formats.

    Source: §1 · p. 2

  • Method

    The study evaluates 3–8-bit llama.cpp GGUF formats from one Llama-3.1-8B-Instruct checkpoint against an FP16 baseline, measuring downstream tasks, perplexity, size, quantization time, and CPU throughput.

    Source: §1 · p. 2

  • Results

    Mid-bit quantization preserves capabilities surprisingly well; 5-bit configurations maintain, and sometimes slightly improve, aggregate downstream scores relative to FP16 while reducing model size, whereas the most aggressive 3-bit setting degrades most clearly.

    Source: §5 · p. 10

  • Takeaways & Limitations

    For Llama-3.1-8B-Instruct, prefer high-performing 5-bit options when memory is limited but quality matters, consider well-tuned 4-bit K-quants for further footprint reduction with modest loss, and treat 3-bit formats as budget-driven compromises.

    Source: §5 · p. 10

Abstract

from arXiv · show

Quantization is a practical technique for making large language models easier to deploy by reducing the precision used to store and operate on model weights. This can lower memory use and improve runtime feasibility on constrained hardware, which is especially relevant for users running models locally. Quantization in llama.cpp enables large language models to run on commodity hardware, but available formats are often evaluated inconsistently, making it hard to choose among schemes. We present a unified empirical study of the llama.cpp quantization on a single modern model, Llama-3.1-8B-Instruct (FP16, GGUF), covering 3-8 bit K-quant and legacy formats. We evaluate downstream task performance across standard reasoning, knowledge, instruction-following, and truthfulness benchmarks, and also measure perplexity and CPU throughput (prefill/decoding) alongside model size, compression, and quantization time. Ultimately, this work is a practical guide for choosing a llama.cpp quantization scheme, helping readers make informed, context-aware decisions for their intended use and resource budget.

1 Introduction

Quantization can make large language models more feasible on constrained hardware, but llama.cpp users face many formats with unclear quality, size, and speed trade-offs. This study compares those formats on Llama-3.1-8B-Instruct using a consistent evaluation pipeline and multiple deployment and quality measures.

1 Introduction

  • Full-precision models can require tens to hundreds of gigabytes of inference memory, while quantization reduces weight and activation precision to make deployment more practical.Prior work reports 2–4× memory and bandwidth cost reductions with 8-bit and 4-bit representations while preserving most predictive performance [1][2][3].Source: §1 · p. 1
  • llama.cpp offers many GGUF weight-only formats, but practical selection advice has largely come from anecdote and perplexity measurements rather than unified comparisons.Its original formats use blocks, while K-quants use superblocks and other techniques intended to improve quality at a given model size.Source: §1 · p. 1
  • Existing evaluations often emphasize perplexity, mix base models and instruction-tuned variants, and provide little unified downstream evidence for choosing a format on a modern Llama model.These gaps leave questions about quality loss between specific formats and the usability of 3-bit K-quants unresolved in a principled, model-specific way.Source: §1 · p. 2
  • The study compares 13 llama.cpp quantization configurations from 3 to 8 bits against an FP16 baseline, using one Llama-3.1-8B-Instruct checkpoint.It measures reasoning, knowledge, instruction-following, truthfulness, perplexity, model size, compression, quantization time, and CPU prefill and decoding throughput.Source: §1 · p. 2
  • A single-model, consistent evaluation pipeline is intended to consolidate fragmented knowledge into quantitative, reproducible evidence for choosing GGUF formats.The study also presents its analysis as a template for future work on other architectures and backends.Source: §1 · p. 2

2 Background & Related Work

The background explains llama.cpp’s weight quantization and format families, then situates the study amid broader quantization evaluations. It identifies the lack of unified, model-specific comparisons of llama.cpp’s concrete GGUF formats as the remaining gap.

2.1 Post Training Quantization of Large Language Models

  • Q4_0 is a symmetric block-wise 4-bit format with a scale and zero-point of zero, whereas Q4_1 stores a scale and offset for affine dequantization.Other basic formats follow the block-wise pattern and differ in bit width and whether they store an offset or minimum.Source: §2.1 · p. 3
  • Under a high-resolution approximation, adding one bit typically reduces modeled quantization-error variance by about fourfold, subject to clipping, heavy tails, and non-uniform error weighting.Block-wise grouping makes the relevant range local rather than global; finer grouping can improve quality at a fixed nominal bit width.Source: §2.1 · p. 3
  • llama.cpp applies post-training, primarily weight-only quantization: weights are packed into integer codes with block metadata, while activations generally remain floating point.Standard formats commonly use blocks of 32 weights; K-quants use 256-weight superblocks with smaller groups and additional scale/minimum metadata.Source: §2.1 · pp. 2–3

2.2 llama.cpp and Its Quantization Schemes

  • llama.cpp supports standard block-wise integer formats and hierarchical K-quants, whose small, medium, and large variants trade compression against fidelity.K-quants use superblocks and may allocate bits or metadata differently across sub-blocks; their effective bit widths can be non-integer.Source: §2.2 · pp. 3–4
  • The community-driven formats and recommendations have developed through informal engineering workflows, leaving no unified, model-specific evaluation of these formats on a modern LLM.The engine targets local inference across CPUs and lightweight GPUs and stores quantized models in GGUF.Source: §2.2 · pp. 3–4

2.3 Existing Quantization Benchmarks

  • Prior quantization studies span methods, models, and tasks, but often emphasize perplexity, aggregate heterogeneous results, or use generic operators rather than concrete llama.cpp GGUF formats.This study fixes Llama-3.1-8B-Instruct and evaluates official llama.cpp formats under one pipeline using perplexity and diverse downstream benchmarks.Source: §2.3 · pp. 4–5

3 Experimental Setup

The experiments use a single Llama-3.1-8B-Instruct GGUF checkpoint and official llama.cpp quantization across nominal 3–8-bit formats. A common evaluation setup measures downstream tasks, perplexity, size, quantization time, and CPU throughput.

3.1 Base Model & Environment

  • All experiments use Llama-3.1-8B-Instruct in GGUF, with F16 as the reference, on a 96-core dual-socket Intel Xeon Platinum 8488C server.The study uses llama.cpp commit b7600 for quantization and perplexity/throughput measurements and lm_eval v0.4.9.2 for downstream scoring.Source: §3.1 · p. 5

3.2 Quantization Configurations

  • The evaluation compares 13 nominal 3–8-bit formats, each produced from the same FP16 GGUF input using the official llama.cpp utility with only the scheme identifier varied.Wall-clock quantization time is measured on the same machine used for inference; size reduction is computed from the FP16 input and quantized output sizes.Source: §3.2 · p. 5

3.3 Evaluation Tasks & Datasets

  • The suite covers GSM8K, HellaSwag, IFEval, MMLU, TruthfulQA, and WikiText-2 perplexity, with throughput measured at pp = 512 and tg = 128.The unweighted Avg is the mean of the five downstream benchmark scores and excludes perplexity; downstream scoring uses lm_eval, while perplexity and throughput use llama.cpp scripts.Source: §3.3 · pp. 5–6

4 Quantization Results and Trade-offs

Across the evaluated schemes, the strongest compression often brings greater quality loss, but the relationship is non-monotonic and task-dependent. The results identify practical trade-offs among aggregate quality, compression, and throughput rather than one universally best format.

4.2 Task Sensitivity Under Quantization

  • On GSM8K, Q3_K_S scores 68.31 versus 77.63 for F16, while Q5_0 scores 79.08; HellaSwag remains comparatively stable across formats.The task-specific patterns differ: IFEval and MMLU also show larger declines at Q3_K_S and smaller fluctuations for most other configurations, while TruthfulQA values vary by format.Source: §4.2 · p. 7

4.1 Overall Accuracy-Compression behavior

  • Avg 65.49 vs. 69.47 for Q3_K_S is the largest aggregate drop, while Q5_0 reaches the highest mean at 69.92.Most 4–8-bit schemes cluster near the baseline, and the compression–quality relationship is non-monotonic.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 Task Sensitivity Under Quantization

  • Q5_0, Q5_1, and Q5_K_S share PPL 7.43 but show different GSM8K and IFEval outcomes, so perplexity alone does not predict downstream behavior.F16 PPL is 7.32 and Q3_K_S PPL is 8.96.Source: §4.2 · p. 7

4.3 Pareto Frontier Analysis: Size Reduction and Performance Tradeoff

  • Q5_0 gives 65.19% reduction with AvgLoss ≈−0.65%; Q4_K_S reaches 70.83% reduction with AvgLoss ≈0.43% when more compression is needed.At greater reductions, Q3_K_L, Q3_K_M, and Q3_K_S remain on the frontier with progressively higher aggregate loss.Source: §4.3 · pp. 7–8, Figure 1 · p. 8

4.4 Throughput Analysis

  • Lower-bit formats deliver the largest token-generation throughput gains, while prompt-processing throughput varies more irregularly across schemes.The throughput comparison measures prompt processing at pp512 and generation at tg128.Source: §4.4 · p. 8, Table 3 · p. 9
  • Q4_K_S combines a nondominated accuracy–compression position with a large generation-speed gain relative to F16; Q3_K_S maximizes throughput and compression but has substantial accuracy loss.The results describe Q3_K_S as a specialized choice for constrained environments rather than a general-purpose setting.Source: §4.4 · p. 8

4.5 Choosing Quantization Scheme In Practice

  • For general use, mid-bit formats keep task performance close to F16; prefer Q6/Q8 when minimizing PPL and downstream drift matters more than memory.Q3_K_S offers the largest reductions and strong speedups but the largest Avg drops and PPL increases; Table 4’s candidates still require target-workload validation.Source: §4.5 · p. 9, Table 4 · p. 10

5 Conclusion

Across this controlled Llama-3.1-8B-Instruct evaluation, 5-bit formats offer a strong quality–compression trade-off, while the most aggressive 3-bit setting shows the clearest degradations. The paper recommends choosing formats with downstream task results in view and extending evaluation to other models and settings.

5 Conclusion

  • In this setting, 5-bit configurations maintain or slightly improve aggregate downstream scores over FP16 while substantially reducing model size; the most aggressive 3-bit setting degrades most consistently.The 3-bit degradations are particularly pronounced on reasoning-heavy evaluations.Source: §5 · p. 10
  • Quantization format matters beyond nominal bit-width: schemes with similar perplexity can diverge on instruction-following and reasoning benchmarks.The paper therefore cautions against using intrinsic metrics alone to select deployment defaults.Source: §5 · p. 10
  • The paper recommends high-performing 5-bit options when memory is limited but quality matters, well-tuned 4-bit K-quants for further footprint reduction, and 3-bit formats as budget-driven compromises.Future work should test additional model families and sizes, hardware targets, and longer-context regimes, and analyze format design choices behind task-specific regressions.Source: §5 · p. 10

A Quantization Timings

Table 5 reports the storage and conversion costs of the evaluated GGUF quantization schemes, complementing the paper’s quality and throughput measurements.

A Quantization Timings

  • Table 5 reports each scheme’s resulting model size in MiB and size reduction relative to the FP16 GGUF input.The reference input is 15,317.02 MiB.Source: A Quantization Timings · p. 12
  • Table 5 also reports the wall-clock time required to quantize each format with the official llama.cpp tooling on the evaluation system.This is the one-time conversion overhead associated with each format.Source: A Quantization Timings · p. 12
  • These storage and conversion measurements complement the main-text quality and throughput results for deployment decisions.The paper identifies storage savings and conversion overhead as potentially decisive constraints in CPU and edge-oriented deployments.Source: A Quantization Timings · p. 12

B Full Quantization Results

Appendix Table 6 provides the benchmark outputs underlying the main results, with task-specific scoring variants and expanded breakdowns. It reports quantization types across reported tasks and metrics, alongside harness settings and score uncertainty.

B Full Quantization Results

  • Table 6 reports mean performance, and where applicable standard error, for each quantization type on every reported task and metric, including alternative scoring variants and subject-level breakdowns.The expanded outputs include GSM8K exact-match criteria, IFEval scoring variants, MMLU subject areas, and TruthfulQA multiple-choice results.Source: B Full Quantization Results · p. 12
  • IFEval reports instruction-level and prompt-level loose and strict accuracy, while MMLU reports overall and subject-group accuracy.TruthfulQA generation uses BLEU-based accuracy and difference; its multiple-choice variants report accuracy.Source: B Full Quantization Results · pp. 12–13
  • Unless otherwise noted, GSM8K used v3 with 5 shots, HellaSwag v1, IFEval v4, MMLU v2, and TruthfulQA v3 with 0 shots; TruthfulQA MC1 used v2.These settings specify the evaluation harness versions and shot counts.Source: B Full Quantization Results · p. 13
  • On GSM8K (FE) EM, Q3_K_S scores 68.31 ± 1.28 versus 77.63 ± 1.15 for the F16 baseline.Table 6 supplies the benchmark outputs underlying the main results.Source: B Full Quantization Results · pp. 12–13, Table 6 · p. 17
  • Scores are reported in percentage points, with both the mean and standard error multiplied by 100.GSM8K reports exact match (EM) with flexible-extract (FE) or strict-match (SM) answer extraction; HellaSwag reports accuracy (A) and normalized accuracy (AN).Source: B Full Quantization Results · p. 12, Table 6 · p. 17
Loading 2601.14277v1…