Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training

1 INTRODUCTION

• Q-PACE models quantization-related loss increase as a sum of per-layer errors weighted by sensitivity coefficients, accurately predicting loss increase during training.This is a training-time analogue of the post-training “linearity theorem” model.Source: §1 · p. 1
• Q-PACE periodically estimates per-layer sensitivities from forward passes and solves a knapsack problem to reassign bitwidths, with negligible memory and marginal computational overhead.The coefficients depend only on model weights and are reasonably stable across iterations.Source: §1 · p. 2
• The evaluated scope includes pretraining models from 350M to 4B parameters and supervised fine-tuning of Qwen3 models, at budgets of 5.0–6.5 bits per weight.The comparisons use state-of-the-art baselines.Source: §1 · p. 2
• Q-PACE consistently improves over existing mixed-precision strategies across model scales, budgets, and formats, achieving the lowest final loss in every evaluated pretraining setting and recovering up to 81% of the uniform NVFP4-to-MXFP8 loss gap.The same trend holds for supervised fine-tuning.Source: §1 · p. 2
• At a fixed precision budget, Q-PACE achieves better final loss than SNIP and the NVIDIA heuristic; at matched loss, it uses fewer bits per model parameter.Figure 1 compares final-loss improvement for a 500M model across methods and budgets.Source: §1 · p. 2, Figure 1 · p. 2
• Sensitivity depends on layer type and depth and evolves more slowly than parameter updates, so about 10 reallocations over a training run suffice to keep overhead low.The paper reports this pattern from analysis of the allocations.Source: §1 · p. 2

2 THE PROBLEM: QAT WITH ADAPTIVE MIXED-PRECISION

• A bitwidth choice technically selects a matrix-multiplication format, and formats with the same nominal bit count can use different value representations.Higher bitwidth typically lowers loss while increasing training and inference cost.Source: §2 · p. 2
• Resource cost is modeled additively across layers, with each layer’s cost determined by its tensor size and format cost.For inference-memory constraints, tensor size is the number of weight elements and format cost includes storage bits such as block scales.Source: §2 · p. 3
• The objective is to minimize loss for a model’s layer-wise bitwidth allocation subject to a maximum model-size constraint.The allocation specifies a bitwidth for each layer from a set of candidate choices.Source: §2 · pp. 2–3
• In QAT, the allocation sequence affects both the final weights and loss, and each step’s precision changes forward-pass loss and backward-pass gradient errors.The desired sequence minimizes final loss while respecting the cost budget at every iteration.Source: §2 · p. 3
• Directly optimizing the allocation sequence is intractable because it involves interactions among layers and training steps.The paper therefore seeks approximations to the loss that make optimization computationally feasible.Source: §2 · p. 3

3 METHOD

• Under zero-mean isotropic quantization noise with finite variance and uncorrelated layer errors, the second-order model weights each layer’s noise MSE by its aggregated curvature.The curvature coefficient is interpreted as one half of the trace of that layer’s Hessian block.Source: §3 · pp. 3–4
• Q-PACE estimates each layer’s sensitivity by adding Gaussian noise to that layer while keeping other layers at high precision, then measuring the loss increase against noise MSE.The calibration dataset should support accurate loss approximation without excessive computational overhead; the study uses analytic MSE estimates.Source: §3 · p. 4
• Given the estimated sensitivities, Q-PACE solves a multiple-choice knapsack problem to minimize predicted loss degradation under the budget.The dynamic-programming implementation has O(nM|β|) time complexity and takes ≲1 second for the models used.Source: §3 · p. 4
• The method periodically recalibrates sensitivities and reallocates precision because estimates require multiple forward passes, while sensitivity patterns remain stable during training.Reallocation periods below 5B training tokens did not yield measurable improvement in the experiments.Source: §3 · p. 5
• Calibration uses forward passes and only one extra noise tensor the size of the largest layer, avoiding the activation and gradient storage required by gradient-based methods.A recalibration costs mn + 1 forward passes over the calibration data, followed by a negligible-cost knapsack solve.Source: §3 · p. 5
• For a 4B model, measured wall-clock overhead was 5% with a conservative update interval of τ = 5k.The paper notes that overhead can likely be reduced by increasing τ.Source: §3 · p. 5

4.1 EXPERIMENTAL SETUP

• In QAT, only Q, K, V, O, and MLP projection layers are quantized; embeddings and the LM head remain high precision, with candidate formats NVFP4, MXFP6, and MXFP8.The global budget is measured as quantized-weight memory divided by the number of quantized parameters; master weights and optimizer state are stored in FP32.Source: §4.1 · p. 5

4.2 FORWARD ERROR IS SUFFICIENT FOR LOSS ESTIMATION

• Forward error alone predicts final test loss well: the modeled estimate has Pearson correlation r = 0.98 whether layer-wise precision applies to both passes or backward matmuls remain uniformly FP4.The experiment uses a 500M model trained on 74B tokens; backward-pass quantization error is not captured by the forward-error metric.Source: §4.2 · p. 6, Figure 2 · p. 6

4.3 PRETRAINING EXPERIMENTS

• Q-PACE achieves lower final test loss than NVIDIA and SNIP across model scales, budgets, and precision sets, recovering up to 81% of the FP4-to-FP8 loss gap.Its loss improvement is 1.5×–32× the NVIDIA allocation’s across the reported settings.Source: §4.3 · p. 6, Table 1 · p. 6
• With a 6.5-bit budget, adding an 8-bit option to {4, 16} raises recovered FP4-to-FP8 gap from 43% to 75% for 350M and from 48% to 81% for 500M models.The 4B experiment also compares test-loss improvement over uniform FP4 across precision options.Source: §4.3 · pp. 6–7, Figure 3 · p. 7

4.4 SUPERVISED FINE-TUNING EXPERIMENTS

• On Qwen3-4B-Base supervised fine-tuning, Q-PACE reaches the lowest held-out loss at every tested budget and beats NVIDIA on downstream mean accuracy, while remaining within SNIP’s noise range.The experiments use Tulu-3 and budgets from 5.0 to 6.5 bits per parameter over three random seeds.Source: §4.4 · p. 7, Table 2 · p. 7

4.5 PERFORMANCE-MEMORY TRADE-OFF

• At a fixed precision budget, Q-PACE forms an empirical Pareto frontier against NVIDIA for both AdamW and Muon, requiring 1 bit/parameter less with AdamW and approximately 0.5 bits/weight less with Muon at matched loss.Across evaluated inference configurations, Q-PACE checkpoints also achieve lower loss at a given inference-memory budget, or comparable loss with less memory. Figure 4 reports 4B test-loss improvement over FP4 with recovered-gap, realized-bit, and precision-option labels.Source: §4.5 · p. 7, Figure 4 · p. 8, Figure 5a · p. 8

Sensitivity by layer type and depth

• In the 4B model’s final allocation, Q-PACE promotes five of six quantized linear layers in the last block to full precision, while K and V projections receive substantial higher-precision budget under both Q-PACE and SNIP.Q-PACE also assigns high precision to late MLP projections, which SNIP does not promote.Source: §5 · p. 8, Figure 6a · p. 9

Sensitivity over training

• Q-PACE’s BF16 allocation changes most between early and middle training, with K and V promotion recurring, Q promotion declining, and MLP promotion growing; SNIP’s selected-layer count stays nearly constant.The sensitivity pattern combines a factor set by layer type and block number with a time-dependent factor.Source: §5 · p. 8, Figure 6b · p. 9

Reallocation frequency

• For the 350M model, more than five reallocations bring no additional final-loss benefit, showing that useful sensitivity updates occur much more slowly than optimization steps.Dynamic reallocation provides benefit beyond finding a better initial allocation, while keeping recalibration overhead marginal.Source: §5 · pp. 8–9, Figure 5b · p. 8

Non-uniform compression

• Non-uniform compression methods include hardware-feedback reinforcement learning, differentiable search, second-order layer scoring, dynamic programming, and evolutionary search.For LLMs, approaches also allocate by sensitivity or account for layer interactions and perturbations.Source: §6 · p. 9

Relation to HIGGS

• Q-PACE builds on HIGGS’s linearity theorem but reallocates during training, covers weights, activations, and gradient multiplications, and uses an analytic 4−b distortion model for microscaling formats.HIGGS allocates once and compresses weights only, whereas Q-PACE’s analytic model avoids per-layer, per-format error estimates.Source: §6 · p. 9

Mixed precision during training

• Training recipes often retain hand-picked components or final blocks in higher precision, while SNIP periodically reallocates LLM layer precision using forward sensitivity and backward-update divergence.Earlier learned-bitwidth methods target final inference precision; Q-PACE instead periodically reallocates during training under a fixed global budget, using forward-pass-only second-order sensitivity.Source: §6 · p. 9

7 CONCLUSION

• Across pretraining from 350M to 4B parameters and Qwen3-4B supervised fine-tuning, Q-PACE outperforms other methods and recovers up to 81% of the NVFP4-to-MXFP8 loss gap at 6.5 bit/parameter.The method probes linear layers and periodically reassigns precision without memory overhead.Source: §7 · p. 10
• The study is limited to a restricted range of model scales and precision formats, and primarily uses the same layer-wise precision in forward and backward passes.Source: §7 · p. 10

A MODELING LOSS UNDER QUANTIZATION NOISE

• The loss approximation assumes a smooth loss, small zero-mean perturbations with finite moments, and uncorrelated perturbations across layers.Under these conditions, the derivation uses a second-order expansion; the first-order term vanishes in expectation.Source: A MODELING LOSS UNDER QUANTIZATION NOISE · p. 16

2 Tr(Hii) we obtain

• For floating-point quantization, the model uses weight-proportional perturbations, yielding diagonal curvature weighted by squared weight magnitude.Source: §2 · p. 17

B MSE ESTIMATION: ANALYTIC VS MEASURED

• The analytic MSE estimate assumes values are approximately uniform within quantization intervals, giving rounding-error variance s^2/12.Source: B MSE ESTIMATION: ANALYTIC VS MEASURED · p. 18
• One shared error curve is used across NVFP4, MXFP6, MXFP8, and BF16 despite format differences; measured tier-specific errors did not improve allocation.Source: B MSE ESTIMATION: ANALYTIC VS MEASURED · p. 18

• r ≥0.97 across both experimental settings and model sizes; modeled error yields a fitted slope approaching one, while measured error yields approximately one half.The comparison evaluates final allocations using sensitivities refitted on final weights.Source: C ON LOSS LINEARITY VS. FORWARD-PASS… · p. 18

D TRAINING SETUP

• The 350M, 500M, and 4B models use Muon for transformer-block weight matrices and AdamW for other parameters; the 800M model uses AdamW alone.Source: D TRAINING SETUP · p. 18, Table 6 · p. 20
• Qwen3-4B-Base is fine-tuned on 10% of Tülu-3; calibration occurs every 128 steps, producing five allocations over 706 training steps.Source: D TRAINING SETUP · pp. 18, 21
• The experiments use NVFP4, MXFP6, and MXFP8 numerics on NVIDIA B200 GPUs; MXFP6 is represented in E4M3 and run through the MXFP8 GEMM path.Thus the experiments do not assess a native 6-bit execution path.Source: D TRAINING SETUP · p. 21

E SNIP WEIGHT-GRAD-ONLY

• SNIP weight-grad-only uses the full-precision layer gradient and measured tier-specific quantization error; the budgeted solver minimizes the resulting first-order loss-change bound.The paper reports that this score-only variant outperformed the full SNIP pipeline in its observations.Source: E SNIP WEIGHT-GRAD-ONLY · p. 22

F Q-PACE HYPERPARAMETER ABLATION

• For the 350M model, increasing noise levels or calibration data improves allocations over the smallest setting, but gains quickly saturate.Figure 8 reports final test-loss improvement over uniform FP4 across m and |D|.Source: F Q-PACE HYPERPARAMETER ABLATION · p. 22, Figure 8 · p. 22
• m ≥4 and |D| ≥64 are sufficient for effective allocations, keeping calibration overhead small.Source: F Q-PACE HYPERPARAMETER ABLATION · p. 22

G ALTERNATIVE SENSITIVITY MODELS

• Alternative sensitivity models include a symmetric approximation relative to uniform-FP4 loss and a quadratic extension with pairwise layer interactions.In the symmetric approximation, the sum covers weights that are not quantized to FP4.Source: G ALTERNATIVE SENSITIVITY MODELS · pp. 22–23
• The default and inversed Q-PACE variants achieve equal improvements within error; measured weight error, gradient MSE, and quadratic Q-PACE improve test loss by 0.472%–0.491%.Cosine similarity, first-order Taylor, and NVIDIA yield smaller gains over FP4.Source: G ALTERNATIVE SENSITIVITY MODELS · p. 23, Table 7 · p. 23
• Across alternative formulations, Q-PACE performance is robust, while the default formulation remains the strongest variant.Source: G ALTERNATIVE SENSITIVITY MODELS · p. 23

H ADDITIONAL SENSITIVITY ANALYSIS

• Q-PACE’s BF16 allocations are non-uniform across layer types and change during training, while SNIP keeps nearly all K and V projections in BF16 and NVIDIA’s allocation is static.Q-PACE consistently assigns high precision to V projections, while high-precision K and O shares vary.Source: H ADDITIONAL SENSITIVITY ANALYSIS · p. 24, Figure 9 · p. 24
• Q-PACE ranks MLP projections as most sensitive, unlike SNIP, which assigns comparatively greater sensitivity to attention projections, especially K.The profiles are comparatively stable after early training; Q-PACE’s score concentrates on MLP up-projections in pretraining and shifts to MLP down-projections in SFT.Source: H ADDITIONAL SENSITIVITY ANALYSIS · p. 24, Figure 10 · p. 24, I BUDGET TRADE-OFF, SENSITIVITY, AND… · p. 27
• Q-PACE’s relative sensitivity generally rises toward later blocks, especially the final block, while SNIP’s is concentrated toward earlier blocks.After an initial transition, Q-PACE’s depth pattern remains remarkably stable for much of training.Source: H ADDITIONAL SENSITIVITY ANALYSIS · p. 25, Figure 11 · p. 25

I BUDGET TRADE-OFF, SENSITIVITY, AND ALLOCATIONS

• During pretraining, Q-PACE concentrates sensitivity on MLP up-projections; in SFT, it shifts to MLP down-projections, while SNIP distributes sensitivity evenly across projection types.Allocations change minimally after initial recalibration; the 4B analysis covers all 168 switchable layers.Source: I BUDGET TRADE-OFF, SENSITIVITY, AND… · p. 27, Figure 15 · p. 28, Figure 16 · p. 29
• At comparable bit counts, Q-PACE achieves greater loss improvement over uniform NVFP4 than SNIP variants and NVIDIA at both pretraining sizes and in SFT.The analysis also reports better loss–precision trade-offs across the evaluated settings.Source: Figure 14 · p. 27, I BUDGET TRADE-OFF, SENSITIVITY, AND… · p. 27

I.1 DOWNSTREAM TASK SCORES

• Downstream accuracy is averaged over three seeds and 24 tasks, counting Natural Questions once; evaluations are zero-shot except for eight-shot chain-of-thought GSM8K.The reported mean is evaluated using the LM Evaluation Harness.Source: §I.1 · p. 37
• The generation and instruction suite reports GSM8K, IFEval, HumanEval, Natural Questions, and TruthfulQA using their respective accuracy or exact-match metrics.The suite appears in Table 8.Source: §I.1 · p. 37, Table 8 · p. 37
• The knowledge and likelihood suite covers ARC-Challenge, ARC-Easy, HellaSwag, PIQA, WinoGrande, LAMBADA, and MMLU, using accuracy.The suite appears in Table 9.Source: §I.1 · p. 37, Table 9 · p. 38
• The probe suite includes BoolQ, COPA, RACE, ASDiv, CommonsenseQA, LogiQA, SAT-Math, SciQ, OpenBookQA, AQuA-RAT, LSAT-AR, LSAT-LR, and Natural Questions.Metrics include accuracy, length-normalized accuracy, and exact match; the results appear in Table 10.Source: §I.1 · p. 37, Table 10 · p. 39