PaperlayerView exact revision on arXiv

Source-linked AI summary

AMQ: Enabling AutoML for Mixed-precision Weight-Only Quantization of Large Language Models

Sangjun Lee, Seung-taek Woo, Jungyu Jin, Changhun Lee, Eunhyeok Park

arXiv:2509.12019v1cs.LGcs.AIcs.CL

TL;DR

Finding a high-performing LLM under a fixed memory budget is difficult because mixed-precision search is combinatorial and evaluation is costly. AMQ combines search-space pruning, a quantization proxy, a quality predictor, and iterative search-and-update. It outperforms BitStack across bit-widths and model scales, retaining up to 99.18% of FP16 average zero-shot performance for a 70B model at 3.5 bits.

  • Problem

    Choosing quantization bit-widths under a fixed memory budget is a combinatorial optimization problem, while format conversion and full validation make black-box search impractical.

    Source: §1 · pp. 1–2

  • Method

    AMQ combines prior-knowledge search-space pruning, a quantization proxy, a quality predictor, and iterative search-and-update to find layer-wise weight-only bit-width configurations.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    AMQ consistently outperforms BitStack across bit-widths and model scales; at 3.5 bits, a 70B model retains up to 99.18% of FP16 average zero-shot performance.

    Source: §4.1 · p. 5

  • Takeaways & Limitations

    AMQ efficiently allocates bit-widths to linear layers at lower precision while addressing real-world constraints.

    Source: §6 · p. 9

    AMQ currently focuses on weight-only quantization; activation quantization for computation-bound challenges remains future work.

    Source: Limitations · p. 9

Abstract

from arXiv · show

To enable broader deployment of Large Language Models (LLMs), it is essential to identify the best-performing model under strict memory constraints. We present AMQ, Automated Mixed-Precision Weight-Only Quantization, a framework that assigns layer-wise quantization bit-widths to optimally balance model quality and memory usage. However, the combinatorial search space, with over 10^{100} possible configurations, makes conventional black-box optimization infeasible. AMQ overcomes this challenge through four key innovations:(1) search space pruning using prior knowledge to exclude unpromising configurations, (2) quantization proxy to bypass costly format conversions during search, (3) quality predictor to minimize evaluation overhead, and (4) iterative search-and-update strategy for fast and stable convergence. By integrating these components, AMQ efficiently explores the quality-efficiency landscape, reaching the Pareto frontier and yielding LLMs that are both compact and high-performing. Our code is available at https://github.com/dlwns147/amq.

1 Introduction

Weight-only quantization can reduce LLM memory use, but accuracy often degrades below 4 bits, motivating the search for the best model under a fixed memory budget. Fine-grained bit-width allocation expands the configuration space, while conversion and evaluation costs make conventional black-box optimization impractical.

1 Introduction

  • Below 4-bit quantization, increased quantization error often causes substantial accuracy degradation, sharpening the fixed-memory question of how to maximize model performance.The cited prior work reports that even 4-bit quantization can preserve model accuracy.Source: §1 · p. 1
  • Layer sensitivity varies across models, modules, and layers, making fine-grained bit-width allocation a possible route to better quality-efficiency trade-offs but greatly enlarging the search space.The paper frames efficient identification of an optimal configuration as a key, largely unexplored challenge.Source: §1 · p. 1
  • The discrete configuration problem is NP-complete, while format conversion can take hours and full validation may require thousands to millions of evaluations per configuration.These costs leave black-box approaches prohibitively expensive despite aggressive evaluation pruning.Source: §1 · pp. 1–2
  • AMQ combines prior-knowledge pruning, a quantization proxy, a quality predictor, and iterative search-and-update to find configurations under fixed memory constraints.The framework aims to maximize accuracy and execution efficiency without expert intervention.Source: §1 · p. 2
  • AMQ consistently outperforms existing methods in accuracy under the same memory constraints and delivers the highest throughput, as illustrated by the memory–accuracy and inference-speed trade-offs.Figure 1 covers average zero-shot accuracy across six tasks and inference speed.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Works

Prior work reduces LLM weight memory with fixed-precision or mixed-precision quantization and offers compression for varying memory targets. AMQ draws on predictor-based AutoML search while adding ideas tailored to quantization.

2.1 Weight-only Quantization

  • Fixed-precision GPTQ and AWQ reduce weights to 3 or 4 bits with limited performance degradation, while mixed-precision methods improve performance but can impede real-world speed through irregular data access.Mixed-precision methods assign different precisions to groups or channels within linear layers based on weight distribution.Source: §2.1 · p. 2

2.2 Model Compression with Varying Targets

  • BitStack supports varied compression targets through low-rank weight storage and residual-block reconstruction, but that reconstruction notably slows inference.LLM-MQ and CMPQ instead use linear- or channel-level mixed precision to match variable target bits, with irregular patterns as a shared issue.Source: §2.2 · p. 2

2.3 AutoML for Neural Architecture Search

  • AMQ draws on neural-predictor research in neural architecture search to reduce search costs, adding quantization-specific ideas needed to make its approach feasible.NAS work includes black-box and differentiable approaches to discrete architecture spaces.Source: §2.3 · p. 2

3 Method

AMQ seeks a Pareto frontier of model quality and memory use while reducing the costs of configuration search. It combines layer-wise search, sensitivity-based pruning, proxy quantization, predicted quality estimates, and iterative updates.

3 Method

  • AMQ frames quantization search as finding a quality-memory Pareto frontier, then selecting its most accurate model within a memory budget.NSGA-II explores Pareto-optimal solutions across multiple objectives, but the required hundreds of thousands of quantization and evaluation runs make direct search costly.Source: §3 · pp. 2–3

3.1 Search Space Design

  • Layer-wise 2-, 3-, or 4-bit assignments provide a hardware-aligned search unit; Llama 2 7B yields 3224 ≈ 10^106 configurations and a group-size-128 range of 2.25–4.25 bits.The paper reports that finer-grained assignments can create meaningless candidates, risk suboptimal convergence, and disrupt hardware-friendly memory access.Source: §3.1 · p. 3

3.2 Space Pruning via Prior Knowledge

  • Layers with sensitivity above twice the median are fixed at 4 bits, leaving the remaining layers active in the search.This calibration-set-based pruning excludes 4, 3, and 5 layers for Llama 2 7B, 13B, and 70B, respectively—1.8%, 1.1%, and 0.9% of linear layers.Source: §3.2 · p. 3

3.3 Quantization Proxy

  • A proxy assembles configurations from precomputed quantized layers, avoiding activation-dependent end-to-end conversion for each candidate; HQQ’s Pareto frontier closely aligns with GPTQ and asymmetric-clipping AWQ.The method precomputes each linear layer at 2-, 3-, and 4-bit precision, then transfers selected bit-width assignments to GPTQ or AWQ for deployment.Source: §3.3 · p. 4

3.4 Quality Predictor

  • An RBF predictor estimates unseen configurations’ quality from a small set of JSD-labeled samples, allowing rapid evaluation of hundreds of thousands of configurations.JSD measures the divergence between quantized-model and original-model logits; approximate Pareto-frontier errors are mitigated by iterative updates.Source: §3.4 · pp. 4–5

3.5 Iterative Search-and-update

  • NSGA-II searches predicted quality and average bit-width, evaluates candidates for true quality, updates the archive, retrains the predictor, and repeats before selecting a configuration that meets the memory constraint.Randomly sampled configurations initialize the archive and predictor; repeated updates refine estimates and solution quality over the fixed search iterations.Source: §3.5 · p. 5

4 Experiments

The experiments compare AMQ with mixed-precision and fixed-precision compression methods across Llama models and memory budgets. AMQ achieves strong quality at low bit-widths and accelerates inference with layer-specific kernels.

4 Experiments

  • The evaluation uses 128 WikiText-2 training samples for calibration and measures perplexity, zero-shot and 5-shot task performance, and inference throughput.Models span Llama 2 7B–70B, with additional results for Llama 3.1, Qwen2.5, and Mistral; AMQ configurations are evaluated using AWQ with group size 128.Source: §4 · p. 5

4.1 AMQ vs. Any-Size Compression

  • At 3.5 bits, AMQ retains up to 99.18% of the FP16 model’s average zero-shot performance on Llama 2 70B and consistently outperforms BitStack across bit-widths and model scales.At 2.5 bits, AMQ has the best average zero-shot accuracy among the compared methods; Table 1 covers Llama 2 7B/13B/70B, perplexity, and zero-shot accuracy.Source: §4.1 · p. 5, Table 1 · p. 6

4.2 Inference Acceleration

  • AMQ achieves up to 2.67× speedup over FP16 on Llama 2 7B and 3.16× on Llama 2 13B by customizing inference kernels for each linear layer’s bit configuration.The reported inference speedups are on the L40S GPU; the method also maintains a small memory footprint for memory-constrained settings such as the RTX 3090.Source: §4.2 · p. 6, Figure 8 · p. 7

4.3 AMQ vs. Fixed-Precision Quantization

  • AMQ consistently matches or surpasses uniform GPTQ and AWQ across bit-widths from 2.25 to 4 bits and across model sizes.For Llama 2 7B, AMQ maintains stable accuracy at 2.35 bits while GPTQ and AWQ degrade sharply at 2.25 bits; on Llama 2 13B, AMQ reaches 96.01% of FP16 average zero-shot accuracy with 3 bits.Source: §4.3 · pp. 6–7

5 Analysis

AMQ reduces search costs, while pruning, repeated-seed evaluation, and layer-wise allocation analyses characterize its search behavior. The analyses link pruning to improved search quality and show consistent allocation preferences across bit-widths.

5.2 Effect of Search Space Pruning

  • Pruning markedly improves search quality on Llama 2 70B and Llama 3.1 70B; without it, Llama 2 70B search leaves configurations near 4.25 bits unexplored.The analysis attributes the issue to outlier linear layers destabilizing training and adding noise to quality predictions, which steers search toward suboptimal points, especially at large scale.Source: Figure 10 · p. 8, Figure 9 · p. 8, §5.2 · p. 8

5.1 Search and Compression Cost

  • On Llama 2 7B, AMQ directly evaluates 10,250 candidates and predicts performance for over 800,000 samples, using a proxy, predictor, and pruning to make search practical.The described search space is approximately 10^100 configurations; the comparison of search and compression time covers AWQ, OmniQuant, BitStack, and AMQ on Llama 2 using A100-80GB GPUs.Source: §5.1 · pp. 7–8, Table 4 · p. 8

5.3 Robustness over Random Seed

  • On Llama 2 7B, the C4-perplexity Pareto frontiers across six random seeds converge consistently across iterations.Figure 11 reports iterations 10, 20, 50, and 200; its data points are shown only when all six seeds discover a sample for that bit-width and iteration.Source: Figure 11 · p. 8, §5.3 · p. 8

5.2 Effect of Search Space Pruning

  • Across calibration sets, sensitive layers repeatedly occur in early self-attention V and early or late MLP Down layers; excluding 0.45–2.14% of candidates has negligible C4 perplexity impact.AMQ converges stably when the sensitivity threshold is not overly strict, and the study adopts twice the median threshold.Source: §5.2 · p. 8, Table 5 · p. 9

5.4 Visualization of Bit Allocation

  • As average bit-width decreases, Query and Key layers receive lower precision first, followed by the MLP Gate layer, while Value layers retain higher precision.The allocation visualization covers Llama 2 13B configurations averaging 2.5, 3.0, 3.5, and 4.0 bits.Source: §5.4 · p. 9, Figure 12 · p. 9

6 Conclusion

AMQ targets model quality under memory constraints through layer-wise mixed-precision weight-only quantization. Its pruning, quantization proxy, and quality predictor are presented as means to reduce search and evaluation costs while allocating bit-widths effectively.

6 Conclusion

  • AMQ selectively excludes low-bit-sensitive outlier layers to prune the search space, uses a quantization proxy to generate models rapidly, and predicts performance for unseen bit-width combinations to reduce evaluation overhead.The conclusion reports that AMQ allocates bit-widths to each linear layer, including at lower precision levels, and outperforms existing baselines under real-world constraints.Source: §6 · p. 9

Limitations

The paper identifies alternative solvers as a possible route to better configurations and leaves them for future work. It also proves that order-equivalent quality measures induce the same Pareto frontier and gives the detailed search algorithm’s inputs.

Limitations

  • Alternative solvers for the NP-complete bit-configuration problem may find better configurations, and investigating them remains future work.The current approach uses a quality-score predictor, a genetic algorithm, and search-space shrinking.Source: Limitations · p. 9

A Proof of Motivation in Section 3.3

B Detailed Algorithm

  • Algorithm 1 takes a search space, 2-, 3-, and 4-bit proxy models, calibration data, sample and iteration counts, and a target bit budget.These are the stated required inputs for the Auto Weight Quantization Search.Source: B Detailed Algorithm · p. 13

C Detailed Experimental Setting

Experiments use up to two A100 80 GB GPUs and specify settings for candidate selection, layer sensitivity, memory accounting, and inference kernels. Across tested NSGA-II crossover and mutation settings, AMQ maintains strong performance, with no single setting clearly superior.

C Detailed Experimental Setting

  • Experiments ran on up to two NVIDIA A100 80 GB GPUs and selected, per target precision, the lowest-score candidate within ±0.005 average bit-width.The same calibration set was used for JSD layer-sensitivity measurement and model-quality evaluation; PB-LLM counts weight memory only, with group size 128 and indexing overhead excluded.Source: C Detailed Experimental Setting · p. 13
  • AMQ used FasterTransformer attention and layer-normalization kernels, TensorRT-LLM for 4-bit linear layers, and AutoGPTQ for 2- and 3-bit linear layers.Search hyperparameters for the Llama 2 family are listed in Table 6.Source: C Detailed Experimental Setting · p. 13, Table 6 · p. 13

D Robustness over NSGA-II Hyperparameter

E Choice of Search Method, Quantization Proxy, Predictor, and Iteration

The appendix explains choices of search method, proxy, predictor, and iteration count. It reports that 100 iterations can still produce competitive models when resources or time are limited, and provides visualizations of searched bit allocations for Llama 2 7B and 70B.

E Choice of Search Method, Quantization Proxy, Predictor, and Iteration

  • NSGA-II is selected to find the Pareto frontier; HQQ is used as the proxy for stronger alignment with actual performance, and RBF predictors for efficiency.The authors report minimal performance differences between MLP and RBF predictors, while alternatives were slower to train or offered negligible improvement.Source: E Choice of Search Method,… · p. 14, Table 9 · p. 14
  • With limited resources or time, 100 iterations still produce competitive models, while more iterations substantially raise search costs for negligible performance improvement.AMQ uses Pareto-superior candidates in its search-and-update procedure; defaults are 200 iterations up to 30B parameters and 250 for larger models.Source: E Choice of Search Method,… · pp. 14–15

F Additional Visualization of Bit Allocation

G Comparison with Other Discrete Structure Search Methods

The section compares AMQ with one-shot and greedy search, lightweight alternatives proposed because exhaustive search over Llama 2 7B’s bit assignments is impractical. AMQ is reported to achieve lower perplexity and higher zero-shot task performance than both heuristics.

G Comparison with Other Discrete Structure Search Methods

  • Llama 2 7B has about 10^106 possible bit-width assignments, making exhaustive grid search impractical.The paper proposes two lightweight search methods in response.Source: G Comparison with Other Discrete… · p. 15
  • One-shot search assigns 4 bits to the most JSD-sensitive layers and 2 bits to the least sensitive; greedy search repeatedly fixes the layer with the smallest JSD-loss increase at 2 bits.Both procedures continue to meet a target average bit-width; the reported cost comparison is for Llama 2 7B/13B on one RTX6000-ADA.Source: G Comparison with Other Discrete… · p. 15, Table 11 · p. 15
  • AMQ outperformed both one-shot and greedy search, achieving significantly lower perplexity and higher zero-shot task performance, and surpassing heuristic methods in optimizing the memory-quality trade-off.Source: G Comparison with Other Discrete… · p. 15

H Additional Experiments

The additional experiments evaluate AMQ against baselines across several LLM families and compare search and quantization approaches. A visualization shows layer-wise bit allocation for Llama 2 70B at different average bit levels.

H Additional Experiments

  • Tables 14–19 evaluate AMQ against baselines on Llama 3.1 8B/70B, Qwen2.5 7B/14B, and Mistral 7B v0.3.These tables cover both any-size compression methods and fixed-precision quantization methods.Source: H Additional Experiments · p. 15, Table 14 · p. 17, Table 15 · p. 18
  • Figure 14 visualizes bit allocations over Llama 2 70B linear layers at different average bit levels, with rows representing Q, K, V, O, Gate, Up, and Down.Numbers at the left indicate the average bits per configuration.Source: Figure 14 · p. 16
  • Tables 12 and 13 evaluate one-shot and greedy search against AMQ, and fixed-precision quantization methods with AMQ, on Llama 2 7B/13B/70B.Table 12 covers Llama 2 7B/13B; Table 13 covers Llama 2 7B/13B/70B.Source: Table 12 · p. 16, Table 13 · p. 17

Related papers

Loading 2509.12019v1…