FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference

Gongwei Lee; Ji Liu; Juncheng Jia; Ji Wu

Source-linked AI summary

FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference

Gongwei Lee, Ji Liu, Juncheng Jia, Ji Wu

arXiv:2608.24945v1Aug 24, 2026cs.LGcs.AIcs.DC

TL;DR

Deploying LLMs on resource-constrained devices requires quantization, but existing methods use uniform bit widths or inadequate sensitivity measures. FAMPWQ combines perturbation-based Fisher sensitivity with PPO-based adaptive allocation, outperforming seven baselines across seven models and five benchmarks, with PPL up to 3.39 smaller.

Problem
LLM quantization needs bit-width allocation that minimizes accuracy degradation under compression constraints, while existing sensitivity metrics do not capture quantization-specific perturbations.
Method
FAMPWQ measures layer-wise quantization sensitivity using perturbation-based Fisher Information and uses PPO to generate adaptive bit-width allocations under a storage budget.
Results
FAMPWQ outperforms 7 baselines across 7 models and 5 benchmarks in PPL, accuracy, and LLM-as-a-judge comparison, with PPL up to 3.39 smaller.
Takeaways & Limitations
FAMPWQ shows particularly strong advantages at the 3-bit compression frontier for memory-constrained LLM deployment.
Takeaways & Limitations
Mixed-precision quantization can reduce inference throughput because heterogeneous bit widths are less compatible with optimized uniform-precision kernels.

Abstract

from arXiv · show

Recent years have witnessed remarkable achievements of Large Language Models (LLMs) in multiple domains, while the excessive resource requirements of LLMs hinder the deployment on resource-constrained devices. Although model quantization stands out as an effective approach, conventional quantization approaches typically incur severe performance degradation due to uniform bit-width or simple heuristic sensitivity evaluation. In this paper, we propose a novel Fisher information-based Adaptive Mixed Precision Weight Quantization approach, i.e., FAMPWQ, which performs layer-adaptive weight quantization for effective LLM inference on commodity GPUs. First, we propose a system model with a novel Fisher information metric to measure the layer-wise sensitivity to quantization. Second, we propose a reinforcement learning-based bit-width allocator in FAMPWQ, which generates an adaptive bit-width allocation strategy based on the Fisher information sensitivity metric. Extensive experiments on 7 models and 5 benchmarks demonstrate that FAMPWQ significantly outperforms 7 baseline approaches in terms of PPL (up to 3.39 smaller), accuracy (up to 6.87% higher), and LLM-as-a-judge comparison (up to 76% win rate).

1 Introduction

FAMPWQ addresses resource-intensive LLM deployment by using quantization-specific Fisher sensitivity to guide layer-adaptive mixed-precision weight quantization. It combines this metric with PPO-based bit-width allocation and outperforms seven baselines across seven models and five benchmarks.

Limitations of prior work: Uniform bit-width methods overlook layer-wise sensitivity, while mixed-precision heuristics based on weight magnitude or raw gradient norms may not reflect quantization-induced degradation.Existing metrics also fail to capture loss-surface geometry under bit-width-specific perturbations because they rely on first-order statistics or curvature at unperturbed weights.
Motivation: LLM layers exhibit heterogeneous quantization sensitivity: attention value projections and MLP down-projections can be orders of magnitude more sensitive than redundant layers.Aggressively quantizing a few sensitive layers can disproportionately degrade model quality, whereas many layers tolerate extreme compression with negligible impact.
Method: FAMPWQ is a weight-only mixed-precision PTQ framework for fixed-memory LLM deployment that uses quantization-perturbation Fisher sensitivity to estimate per-layer degradation.The framework uses a low-cost proxy optimizer to allocate layer bit-widths under a storage budget.
Sensitivity metric: The Fisher metric injects quantization-simulating perturbations and measures the resulting Fisher shift to capture layer-wise sensitivity to quantization loss.This targets quantization-specific loss geometry rather than magnitude or gradient proxies.
Bit-width allocation: PPO guides adaptive layer-wise bit-width allocation under a storage constraint using each layer’s quantization-perturbation Fisher sensitivity.The allocator searches mixed-precision strategies for layer-wise LLM quantization.
Experiments: PPL was up to 3.39 smaller, accuracy up to 6.87% higher, and LLM-as-a-judge win rate up to 76% across seven models and five benchmarks.FAMPWQ significantly outperformed seven baseline approaches while maintaining compatibility with methods such as GPTQ and AWQ.

2 Related Work

Related work improves post-training quantization through error reduction, outlier correction, calibration, adaptive clipping, and bit balancing, while mixed-precision methods pursue sensitivity- or hardware-aware bit-width allocation. These approaches motivate adaptive allocation strategies for low-bit LLM quantization.

Related Work: Post-training quantization methods reduce compression error through weight-activation redistribution, calibration, adaptive clipping, and bit-balance strategies.Examples include SmoothQuant (Xiao et al., 2023), GPTAQ (Li et al., 2025), OmniQuant (Shao et al., 2024), and ABQ-LLM.
Related Work: Mixed-precision and search-based methods allocate bit-widths using expert granularity, activation guidance, reinforcement learning, or combinatorial optimization.DeepSeek-V4 models use expert-aware FP4/FP8 quantization, while AMQ (Lee et al., 2025), HAQ (Wang et al., 2019), COPAL (Malla et al., 2024), and BitWeaver explore adaptive allocation.

4 FAMPWQ Methodology

FAMPWQ measures each layer’s quantization sensitivity by the Fisher-information change caused by its exact b-bit rounding perturbation, then uses PPO to adaptively allocate layer-wise bit-widths under a compression target.

Fisher-information sensitivity: The perturbation is generated from b-bit quantization so that θ_l + δθ_l = Q_b(θ_l), making it the exact additive rounding noise rather than arbitrary noise.This targets the specific quantization noise incurred at the selected bit-width, unlike uniform or magnitude-proportional perturbations.
Fisher-information sensitivity: FAMPWQ measures layer sensitivity as the Euclidean-norm variation between diagonal Fisher information vectors computed before and after quantization perturbation.The perturbation is applied to one target layer while all other layers retain their original parameters, and the diagonal FIM approximates the full matrix.
Adaptive bit-width allocation: FAMPWQ formulates bit-width allocation as loss minimization with penalties and rewards tied to whether the target compression rate is achieved.The original combinatorial optimization problem is transformed into a single loss function, with constant terms for failing to meet or exceeding the compression target.
Adaptive bit-width allocation: PPO allocates bit-widths layer by layer: an actor proposes strategies, while a critic and reward-based environment provide feedback for stable policy improvement.The actor uses a clipped surrogate objective, and the critic is trained using the squared temporal-difference advantage.
Inference procedure: During inference, the allocation starts from the highest selectable bit-width and the actor sequentially generates each layer’s bit-width to form the final quantization strategy.After L steps, the generated strategy Q_L is used to quantize the LLM.

5 Experiments

Across 7 models and 5 benchmarks, FAMPWQ consistently improves perplexity, zero-shot accuracy, and LLM-as-a-judge win rates over seven baselines, while providing throughput and preprocessing benefits. Ablations show that Fisher-based sensitivity and reinforcement-learning allocation materially improve quantization quality and remain robust to hyperparameter choice.

PPL evaluation: FAMPWQ outperforms all baselines in PPL, reducing average PPL by up to 2.37 over GPTQ and 0.40 over AWQ on LLaMA-7B at 3-bit quantization.PPL results are averaged over three independent calibration-sampling runs; FAMPWQ also achieves 5.81 on WikiText-2 and 10.34 on PTB for LLaMA-7B at 4 average bits.
Zero-shot accuracy: At 3-bit quantization on Qwen2.5-7B, FAMPWQ exceeds RTN and GPTQ by 6.87% and 6.69% in average zero-shot accuracy, respectively.Figure 5 reports consistent improvements across five reasoning tasks and indicates better preservation of general reasoning ability under aggressive compression.
LLM-as-a-judge comparison: FAMPWQ achieves win rates of 76% against RTN, 69% against OWQ, 64% against GPTQ, 72% against GPTQv2, 61% against AWQ, and 54% against OmniQuant.The comparison uses GPT-3.5-turbo, 80 questions, bidirectional comparisons, and 160 trials per comparison; ties are excluded.
Inference efficiency: FAMPWQ improves throughput over FP16 by up to 42%, OWQ by up to 69%, and AMQ by up to 28%, but remains slower than uniform low-bit AWQ.AWQ has the highest absolute throughput at 2.44× on 7B and 2.10× on 13B because it exploits uniform 4-bit kernels.
Allocation and efficiency ablations: The RL-based allocator lowers average PPL by up to 1.50 versus Greedy, 1.33 versus Bayesian optimization, 1.90 versus simulated annealing, and 0.45 versus genetic search.The b=4 perturbation form also beats δ1 by up to 0.56 and δ2 by up to 0.23, while preprocessing remains below 1 GPU-hour and RL search takes under 5 minutes.
Sensitivity analysis: The FIM-based sensitivity metric correlates with Oracle sensitivity at r=0.91 and r=0.88, limiting ΔPPL to +0.42 versus +0.85 for weight magnitude and +1.19 for random allocation at 3.5-bit average.The correlations are reported on LLaMA-7B and Qwen2.5-7B, with p < 10^-14 and p < 10^-17, respectively.

6 Conclusion

FAMPWQ is a Fisher information-based adaptive mixed-precision weight quantization approach that captures layer-wise quantization sensitivity and allocates bit-widths adaptively. Across 7 models and 5 benchmarks, it outperforms 7 baselines in PPL, accuracy, and LLM-as-a-judge comparison.

6 Conclusion: The proposed method is a novel Fisher information-based Adaptive Mixed Precision Weight Quantization approach for effective LLM inference.
6 Conclusion: Up to 3.39 smaller PPL, 6.87% higher accuracy, and 76% win rate against 7 baselines across 7 models and 5 benchmarks demonstrate FAMPWQ’s effectiveness.The win rate refers to LLM-as-a-judge comparison.
6 Conclusion: FAMPWQ combines a perturbation-based Fisher Information metric for layer-wise quantization-specific sensitivity with a PPO-based method for adaptive bit-width allocation.The allocation method efficiently generates the adaptive bit-width strategy.

Limitations

FAMPWQ has limitations in inference throughput, quantization scope, and model coverage. Mixed-precision execution targets memory-constrained deployment rather than peak tokens-per-second, while weight-activation quantization and broader model types remain outside the current scope.

Limitations: Mixed-precision quantization can reduce inference throughput because heterogeneous bit-widths are less compatible with optimized uniform-precision kernels.FAMPWQ therefore focuses on memory-constrained deployment rather than peak tokens-per-second.
Limitations: FAMPWQ currently targets weight-only quantization (WxA16), leaving joint weight-activation quantization to future work.
Limitations: The experiments focus on dense Transformer models, limiting the reported evaluation scope.

A Appendix · A.1 Explanation of Notations

Appendix A.1 states that the paper’s notation meanings are summarized in Table A.1.

A.1 Explanation of Notations: Table A.1 summarizes the meanings of the notations used throughout the paper.

A.2 Preliminary

This section presents quantization preliminaries and introduces the calculation of Fisher information.

A.2 Preliminary: The section establishes the quantization preliminaries.
A.2 Preliminary: It introduces how Fisher information is calculated.

A.2.1 Quantization Preliminary

This section introduces uniform and non-uniform quantization for LLM weights and formalizes quantization, dequantization, and global-loss optimization. FAMPWQ uses symmetric uniform quantization with zero-point fixed at 0.

Quantization Preliminary: Quantization is classified as uniform, using equally spaced finite intervals, or non-uniform, using intervals whose spacing and lengths can vary.For b-bit integer representations, uniform quantization uses 2^b intervals.
Quantization Preliminary: The quantization process maps an LLM weight tensor W to a quantized tensor WQ and recovers it as fW through dequantization.Uniform quantization applies a rounding-to-nearest operation to a scaled input, with bit-width, scale, zero-point, and clipping parameters.
Quantization Preliminary: FAMPWQ uses symmetric uniform quantization, which restricts the zero-point or offset value to 0.Uniform quantization may otherwise be symmetric or asymmetric according to the sign of the mapping space.
Quantization Preliminary: The quantization approach optimizes the global loss associated with the input X of the corresponding LLM layer.The global-loss objective is defined over the layer’s input.

A.2.2 Fisher Information

Fisher information measures parameter influence through output sensitivity and the likelihood function, enabling layer-wise importance estimation for accuracy-preserving LLM compression. FAMPWQ uses the empirical FIM and its diagonal representation to make this analysis practical.

A.2.2 Fisher Information: Fisher information quantifies parameter importance by measuring how sensitive model outputs are to parameter noise.The Fisher Information Matrix is defined as the expected outer product of score vectors.
A.2.2 Fisher Information: The empirical FIM approximates the expected FIM using samples from the validation dataset D.The approximation is defined using N validation samples (Kunstner et al., 2019).
A.2.2 Fisher Information: Larger FIM values identify more influential parameters that should be preserved during inference and compression.FIM supports layer-wise parameter importance evaluation for preserving accuracy while compressing LLMs (Singh and Alistarh, 2020; Liu et al., 2021).
A.2.2 Fisher Information: Because full FIM computation is expensive for large gradient matrices, the method represents FIM with its diagonal vector.The diagonal representation uses the identity matrix and element-wise multiplication to match the gradient-matrix dimensions.

A.3 Pearson Correlation Analysis … A.7.13 Qualitative Generation Example at 3-bit

Across its supplementary analyses, FAMPWQ validates Fisher-based sensitivity, adaptive bit allocation, stable low-bit performance, and deployment benefits, while preserving generation quality better than uniform baselines under aggressive compression. These results span correlation, ablation, robustness, efficiency, memory, modeling assumptions, and qualitative evaluation.

A.3 Pearson Correlation Analysis; A.5 Visualization of Sensitivity; A.7.2 PPL with Diverse Quantization Bit-width: FIM reliably tracks quantization sensitivity and exposes substantial layer diversity, enabling FAMPWQ to preserve sensitive input/output-adjacent layers while compressing less-sensitive layers more aggressively.Pearson analysis finds R > 0.5 with P < 0.0001, while layer-wise PPL tests show up to 0.115 PPL reduction for MLP down-projections and up to 0.047 for attention value projections.
A.4 Perturbation Strategy Ablation; A.6 Reinforcement Learning (RL)-based Network Training: The canonical quantization perturbation δ3 at b=4 outperforms magnitude-proportional and Bernoulli-masked alternatives, while the RL allocator generates layer-wise bit-width strategies through actor–critic updates.The alternatives are dominated by δ3; training initializes all layers at maximum bit-width, generates per-layer actions, computes loss and rewards, and updates actor and critic networks.
A.7.1 Calibration-Size Robustness: With only 32 calibration sequences, Fisher rankings retain 0.996 Spearman correlation and 97.1% top-10% overlap with the 256-sequence reference, while only 2.68% of assignments change.At the 128-sequence setting used in the main experiments, the allocation difference decreases to 1.34%, demonstrating stable sensitivity ranking and budget-constrained allocation.
A.7.3 Diverse Average Quantization Bit-width; A.7.12 Accuracy-per-Byte Analysis: FAMPWQ supports average bit-widths from 3.1 to 4.5 with decreasing PPL and improves accuracy-per-byte over AWQ by +0.65 percentage points at 4-bit and +1.55 points at 3-bit.The corresponding 4-bit and 3-bit storage costs are 5,197 MB and 4,414 MB, respectively.
A.7.4 Comparison of Time Consumption; A.7.6 Storage reduction: FAMPWQ’s quantization time is competitive, preprocessing remains below 70 minutes for 14B models, and it reduces storage relative to OWQ without extra metadata.FAMPWQ can be up to 37% faster than GPTQ, 61% faster than GPTQv2, 35% faster than OWQ, and 66% faster than OmniQuant; storage savings versus OWQ are 1%–3% of original model size.
A.7.5 Results of LLaMA2, Qwen2.5 and Mistral-7B-v0.1 models: FAMPWQ achieves the best 4-bit PPL across three LLMs and three benchmarks, while also attaining leading zero-shot accuracy on Qwen2.5 and LLaMA2-chat models.On Qwen2.5-7B, average PPL is 10.81, up to 2.86 lower than RTN; on Qwen2.5-14B, it is 9.15, 1.41 lower than RTN. Accuracy gains reach 4.55% on Qwen2.5 and 2.87% over RTN on LLaMA2-13B-chat.
A.7.8 Packed-Deployment Memory Accounting; A.7.9 Bit-width Allocation Visualization: For packed Llama-2-7B inference, 3-bit packing saves at least 9.605 GiB in tensor-accounted memory, while FIM-based allocations differ structurally from weight-only metrics by assigning higher precision to loss-sensitive modules.The estimate is analytical rather than a measured runtime peak and excludes activation, framework, allocator, and fragmentation effects.