When Lower Reconstruction Loss Hurts: Distributionally Robust Refinement for Low-Bit LLM Quantization

1 INTRODUCTION

• 18/32 AWQ weight modifications increased same-data PPL despite lower reconstruction loss; on held-out WT2, PPL rose for 413/560 AWQ (73.8%) and 264/560 GPTQ (47.1%) lower-loss replacements.Each evaluation changed one linear layer while keeping the other weights fixed.Source: Figure 1 · p. 2, §1 · p. 2
• On held-out C4, further minimizing reconstruction loss raised PPL by 0.69%, while DRQ lowered it by 0.32% despite slightly higher calibration loss.Both PPL changes are relative to the initial quantized model.Source: Figure 1 · p. 2, §1 · p. 3
• The analysis identifies when a calibration-favored quantized weight matrix can have higher reconstruction loss than another under constrained changes in input-activation distributions.The introduction connects the mismatch to input activations that may differ between calibration and new data.Source: §1 · pp. 2–3
• DRQ minimizes worst-case reconstruction loss over constrained activation distributions by updating weight integer codes while preserving quantization parameters and inference operators.The refinement is offline and adds no inference overhead.Source: §1 · p. 3

2.1 WEIGHT-ONLY QUANTIZATION ON A FIXED GRID

• A b-bit uniform quantizer represents each weight with an integer code, while the scale and zero point determine its corresponding quantized value.Group-wise quantization assigns each consecutive group in a row its own shared scale and zero point.Source: §2.1 · p. 3
• With the quantization parameters fixed, changing integer codes is the way DRQ refines quantized weights on the existing grid.The reconstruction formula makes each quantized weight depend on its integer code.Source: §2.1 · p. 3

2.2 RECONSTRUCTION ON CALIBRATION DATA

• Reconstruction loss is the squared difference between quantized and full-precision layer outputs on calibration activations, summed over output channels and averaged over tokens.Figure 2 illustrates a code refinement that increases WT2 reconstruction loss by 1.78% while reducing C4 reconstruction loss by 15.11%, relative to the initial weights.Source: §2.2 · p. 3, Figure 2 · p. 4

3.1 RECONSTRUCTION UNDER CHANGING INPUT DISTRIBUTIONS

• The analysis represents new-data activation second moments as H + Δ, where H comes from calibration data and Δ captures the difference.Restricting Δ to be positive semidefinite means expected reconstruction loss can increase or remain unchanged.Source: §3.1 · p. 4
• The distribution-change budget tr(CΔ) ≤ α limits permitted changes, with C determining how much change is allowed along each input direction.For a rank-one change Δ = αuu⊤ at the full budget, the direction is normalized by u⊤Cu = 1.Source: §3.1 · p. 5
• Theorem 3.1 gives an exact condition for whether one weight matrix with lower calibration loss remains no worse under every permitted distribution change.If the condition fails for a normalized direction, a permitted change along that direction makes its expected loss higher than the comparator's.Source: §3.1 · p. 5

3.2 ROBUST RECONSTRUCTION OBJECTIVE

• DRQ minimizes each matrix's largest reconstruction loss over the constrained distribution changes, combining calibration loss with its worst-case increase.The equivalent objective reduces the inner maximization to the input direction with the largest squared output difference under the same budget.Source: §3.2 · pp. 5–6
• The matrix C = (1 − β)H + βI allows greater increases in directions whose activations were small during calibration while keeping changes bounded in every direction.At β = 1, directions with the same Euclidean norm are treated equally.Source: §3.2 · p. 6

3.3 REFINEMENT ON A FIXED QUANTIZATION GRID

• DRQ refines one Transformer block at a time, changing integer codes while keeping bit width, grouping, scales, and zero points fixed.Its offline search ranks code updates, forms candidate sets, and selects candidates that reduce the robust loss; inference operators remain unchanged.Source: §3.3 · p. 6

4.1 EXPERIMENTAL SETUP

• Evaluation spans dense Llama and Qwen3 MoE models, using WT2/C4 perplexity and seven-task zero-shot accuracy against six PTQ baselines.Source: §4.1 · pp. 6–7

4.2 DENSE-MODEL RESULTS ACROSS SIZES AND BIT WIDTHS

• DRQ improves average task accuracy across GPTQ and AWQ at every evaluated model size and bit width, while reducing perplexity in most cases.For Llama-3.1-70B with GPTQ W2, C4 PPL falls from 31.77 to 27.11 and accuracy rises from 48.23% to 54.56%.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 EXTENSION TO MIXTURE-OF-EXPERTS MODELS

• For Qwen3-30B-A3B at W3A16, DRQ lowers WT2 and C4 perplexity and improves accuracy on five of seven downstream tasks.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 VERSATILITY ACROSS SIX PTQ METHODS

• Across six PTQ methods on Llama-3.2-3B W3, DRQ lowers WT2 and C4 perplexity and raises average accuracy; ParoQuant remains best on all three metrics.Source: Figure 3 · p. 8, §4.4 · p. 8

4.5 CALIBRATION SIZE AND THE CHOICE OF OBJECTIVE

• As calibration size grows, rec-only reduces reconstruction loss further than GPTQ but can worsen PPL by orders of magnitude; DRQ keeps loss near GPTQ’s while lowering PPL on both datasets.The comparison uses GPTQ W4 on Llama-3.2-3B-Instruct, varying calibration sets from 16 to 256 sequences.Source: Figure 4 · p. 8, §4.5 · p. 8

4.6 EFFICIENCY OF DRQ

• On Llama-3.1-8B W3, refinement averages 6.01 minutes with GPTQ and 5.96 minutes with AWQ across 20 configurations.The timed loop quantizes successive Transformer blocks and includes calibration-data forward passes.Source: §4.6 · p. 9

• PTQ work addresses outliers through scaling, clipping, rotations, and transformations, while quantization approaches also use adaptive rounding or trellis coding [3, 7].GPTQ applies second-order error compensation during rounding [Frantar et al., 2023].Source: §5 · p. 9
• Other methods modify reconstruction objectives, accounting for preceding-layer errors, weighting loss with final-model gradients, or matching token probabilities [5, 6].The passage also describes CoreQ’s layer-wise correction strength and LFQ’s application to the final Transformer block.Source: §5 · p. 9
• In DRO, the uncertainty set determines which distribution shifts the objective protects against; excessive freedom can make solutions conservative or amplify outliers.The surveyed approaches include moment constraints, divergence bounds, transport distances, and group-based or sample-reweighting methods.Source: §5 · p. 9

6 CONCLUSION

• DRQ combines calibration reconstruction loss with a worst-case directional error term to refine existing integer codes, improving perplexity and average task accuracy without extra inference overhead.The conclusion reports gains across multiple PTQ methods and dense and mixture-of-experts models, while preserving the quantization grid, scales, and zero points.Source: §6 · p. 9

B THEORETICAL RESULTS AND PROOFS

• The appendix develops the analysis and objective with the full-precision weights and quantization grid fixed, defining H as the calibration-input activation second-moment matrix.H may be computed from samples or from their distribution, as specified in the appendix.Source: B THEORETICAL RESULTS AND PROOFS · p. 14

B.1 THE ROLE OF THE QUANTIZATION GRID

• When full-precision W is representable on the quantization grid, it minimizes reconstruction loss for every finite-second-moment input distribution, uniquely when H ≻ 0.Without a quantization constraint, retaining W gives zero reconstruction loss for any input distribution.Source: §B.1 · pp. 14–15
• Excluding W from the grid permits different minimizers, and a fixed-grid example shows calibration-optimal weights can have higher reconstruction loss on a new input.The notation summary identifies α as the distribution-change budget and the weight on worst-case directional error, and β as the weight on the identity matrix in C.Source: A NOTATION · p. 14, Table 3 · p. 14, §B.1 · p. 15

B.2 WHY MORE CALIBRATION DATA MAY NOT RESOLVE THE MISMATCH

• When expected reconstruction and evaluation losses favor different grid weights, more same-distribution calibration samples can leave exact empirical minimizers with a fixed evaluation-loss disadvantage.The theorem allows evaluation loss to measure model performance on the calibration distribution or reconstruction loss on new activations.Source: §B.2 · p. 15

B.3 WHEN INPUT CHANGES REVERSE THE RECONSTRUCTION LOSS COMPARISON

• A lower-calibration-loss matrix remains no worse under every allowed distribution change exactly when its original loss gap is at least αe12(C).The condition compares layer-output reconstruction losses under the specified positive-semidefinite changes and trace budget.Source: §B.3 · p. 16
• If that condition fails, a rank-one distribution change satisfying the constraints makes the initially lower-loss matrix have higher expected reconstruction loss.The change is constructed from a leading eigenvector of the transformed difference matrix.Source: §B.3 · p. 16

B.4 A FIXED-GRID EXAMPLE OF HIGHER LOSS ON A NEW INPUT

• For any ε > 0, the fixed-grid example has a unique calibration-loss minimizer with lower loss than an alternative but squared error s² versus zero on a unit-length new input.Both quantized vectors share a fixed scale, and the construction uses a trace-one calibration second moment.Source: §B.4 · p. 17

B.5 WORST-CASE RECONSTRUCTION LOSS

• The worst-case loss increase is attained by a rank-one distribution change along the largest-eigenvalue direction, which depends on both the weight difference and C.DRQ recomputes this direction before accepting proposed weights.Source: §B.5 · p. 18

B.6 COMPARING REFINED AND INITIAL QUANTIZED WEIGHTS

• A negative exact Γα(cW1, cW0) certifies that cW1 has strictly lower loss than cW0 throughout the constrained second-moment set.An estimated negative value alone is insufficient for this guarantee because a Rayleigh quotient can underestimate the largest eigenvalue.Source: §B.6 · p. 19

B.7 EQUIVALENT OPTIMIZATION OVER INPUT DIRECTIONS

• The semi-infinite directional formulation preserves the minimum and minimizing weight matrices for every α ≥ 0.It has finitely many optimization variables but one constraint for each of infinitely many input directions.Source: §B.7 · p. 19

B.8 A RECONSTRUCTION BOUND USING WASSERSTEIN DISTANCE

• For distributions within a weighted Wasserstein distance, the reconstruction bound is an upper bound, unlike the equality under DRQ’s distinct second-moment constraints.The bound combines calibration reconstruction loss with DRQ’s worst-case directional error.Source: §B.8 · p. 20, §2 · p. 20

2. Taking the infimum

• Taking the infimum in the Wasserstein derivation yields the stated reconstruction-loss bound for distributions within the specified distance.The derivation applies the L2 Minkowski inequality and then takes the supremum over the permitted distributions.Source: §2 · p. 20

C ALGORITHMIC DETAILS

• DRQ refines each PTQ-quantized linear layer while keeping bit width, grouping, scales, and zero points fixed; proposed code changes must pass loss checks on search and checking subsets.It estimates the worst-case direction again for each proposal and rejects proposals that fail the reconstruction tolerances or numerical checks.Source: C ALGORITHMIC DETAILS · p. 20

D.1 DETAILED DENSE LLAMA RESULTS

• Average task accuracy improves in every reported comparison with the initial quantized model.Tables 4–7 report accuracy on each of seven downstream tasks before and after refinement.Source: §D.1 · p. 21, Table 4 · p. 21, Table 5 · p. 22
• Tables 4–7 cover per-task accuracy for Llama-3.2-1B, Llama-3.2-3B, Llama-3.1-8B, and Llama-3.1-70B Instruct models.Source: Table 4 · p. 21, Table 5 · p. 22, Table 6 · p. 22
• 6.33 percentage points: DRQ raises average accuracy for the 70B model quantized with GPTQ at W2, improving all seven task scores.Table 7 reports accuracy for Llama-3.1-70B-Instruct.Source: §D.1 · p. 21, Table 7 · p. 23

D.2 RESULTS ACROSS PTQ METHODS

• For GPTQ and AWQ, accuracy improves on six of seven tasks, while RTN’s gains on three tasks outweigh declines on four others.Source: §D.2 · p. 23
• Across six selected comparisons, DRQ lowers both WT2 and C4 perplexities and raises average task accuracy.GPTQ and AWQ improve on six of seven tasks; RTN gains on PIQA, BoolQ, and ARC-Easy outweigh declines on its other four tasks.Source: Table 8 · p. 23, §D.2 · p. 23
• 1.28 percentage points: at W3, QuaRot followed by GPTQ raises mean accuracy from 68.44% to 69.72%, improves all seven tasks, and lowers WT2 and C4 PPL.The evaluation uses Llama-3.1-8B-Instruct with group size 128.Source: §D.2 · p. 23, Table 9 · p. 24

D.3 CALIBRATION TOKENS AND EXPERT REFINEMENT

• 4,794 of 18,624 linear layers (25.74%) receive fewer than the 1,024 activation vectors used for full search and checking subsets; 600 receive at most two tokens.The one-, two-, and zero-token layers belong exclusively to experts.Source: §D.3 · p. 24, Figure 6 · p. 24
• With at least three tokens, DRQ forms separate proposal and checking subsets; with one or two it uses all available vectors for search, and with none it uses RTN initialization and identity H and C.Layers with fewer than three tokens lack separate checking subsets; the procedure adds no artificial tokens and does not change routing.Source: §D.3 · p. 24
• 266 of 399 one- or two-token layers are modified (66.7%), versus 8,657 of 18,024 layers with at least three tokens (48.0%).The low-token group contributes 3.67% of code changes; DRQ changes 0.01815% of the counted weights overall, and all zero-token layers retain their initial codes.Source: §D.3 · p. 24, Figure 6 · p. 24

D.4 PARAMETER SENSITIVITY

• At α = 0.3, both GPTQ and AWQ improve all three metrics across all five β values; raising α to 1 worsens WT2 PPL in three GPTQ and two AWQ configurations.The favorable settings depend on the base PTQ method, and a stronger directional-error penalty does not consistently improve performance.Source: Figure 7 · p. 25, §D.4 · p. 25

E.1 REFINEMENT TIME ACROSS PTQ METHODS

• Table 10 reports DRQ refinement time for Llama-3.2-3B W3; its timing columns exclude base-method quantization and hyperparameter search.The integration-inclusive timing also covers surrounding code that calls DRQ or passes stored calibration activations; most methods were remeasured on one A800 80GB GPU.Source: Table 10 · p. 26, §E.1 · p. 26

E.2 SCALING WITH MODEL SIZE AND BIT WIDTH

• At W3, GPTQ refinement takes 2.24, 4.05, 5.95, and 28.05 minutes for 1B, 3B, 8B, and 70B Llama models; AWQ takes 2.00, 3.90, 5.40, and 28.96 minutes.Across 1B–70B models and base PTQ methods, W2 takes longer than W3 and W4, including comparisons using the same seeds.Source: Figure 8 · p. 26, §E.2 · p. 26

E.3 SEARCH COMPLEXITY AND DEPLOYMENT

• Scoring all single-code changes costs O(mdnd + mdB) per round; shortlisting k = 8 changes per output row limits pair scoring to at most 28 candidate pairs per row.Computing shortlisted pairs’ activation inner products costs O(mk^2nd).Source: §E.3 · pp. 26–27
• DRQ retains at most one update per output row, checks p ≤ 6 prefixes, and bounds direct reconstruction checks by O(pmd(nd + nc)).The search uses at most 22 numerical solves per linear layer, with early rejection and stopping reducing that number.Source: §E.3 · p. 27
• DRQ’s computation occurs during offline refinement; it preserves bit width, quantization grid, dequantization rule, and parameter layout, adding no inference operators or stored quantization parameters.The appendix measures resulting latency, throughput, and memory use.Source: §E.3 · p. 27

E.4 INFERENCE OVERHEAD

• −0.80% to +0.82%: prefill and decoding times change across six workloads after refinement, while model tensors occupy the same amount of memory.The benchmark uses Llama-3.1-8B-Instruct at W4A16, group size 128, on one A800 80GB GPU.Source: §E.4 · p. 27, Table 11 · p. 28