PaperlayerView exact revision on arXiv

Source-linked AI summary

From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers

Nafiseh HosseinpourFardi, Negar Alihadi, Mahmoudreza Babaei, Milad Hosseini, Adrian Weller

arXiv:2609.34866v1cs.LGcs.AI

TL;DR

PTQ commonly reconstructs weights matrix by matrix, leaving open whether that objective captures attention behavior or the combined effects of Q, K, and V errors. JAB uses a joint attention loss for quantized-weight refinement and bit allocation; it works for attention-only Mistral-7B, while role-aware allocation performs better beyond attention.

  • Problem

    Most PTQ pipelines use a weight-reconstruction objective distinct from the criterion that sets bit widths, leaving whether one objective can do both untested.

    Source: §1 · p. 2

  • Method

    JAB applies one joint QKV attention loss to quantized-weight refinement and Hessian-trace-based block scoring for knapsack bit allocation.

    Source: §1 · p. 2

  • Results

    At 3 bits, JAB recovers 77–90% of the uniform-GPTQ-to-full-precision gap on attention-only Mistral-7B, but role-offset allocation beats it on the full model.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Beyond attention, matrix role provides a stronger allocation signal than the curvature criterion computed, and attention refinement can recover outputs without preserving weight proximity.

    Source: §1 · p. 3, §7 · p. 9

    At exactly 3 bits on GPT-2, the additive allocation objective can put a unit below 3 bits and miss the low-bit perplexity cliff; JAB's block trace also varies by depth rather than matrix role.

    Source: §4 · p. 5, §6.3 · p. 8

Abstract

from arXiv · show

Most post-training quantization pipelines fit each weight matrix to its pretrained counterpart, one matrix at a time. Whether that proxy tracks what an attention block actually computes, or how errors in the Q, K and V projections compound inside the softmax, is rarely checked. We write the objective on the attention output instead, over all three projections at once, and reuse it throughout the pipeline. JAB defines one scalar loss over the joint Q, K, V weights of a block, evaluated against the block's real causally-masked attention output, and uses it twice: to fit the quantized weights (GPTQ warm start, then STE with learnable scales), and to score the block for a multiple-choice knapsack allocation. On attention-only quantization of Mistral-7B this works. At 3 bits JAB recovers 77-90% of the gap between uniform GPTQ and full precision, and its sensitivity estimate tracks an oracle costing 73 forward passes to within a fraction of a point. It stops working once MLP layers enter the allocation. A role-aware offset rule needing no sensitivity estimate at all beats JAB on GPT-2's MLP and on the full Mistral-7B model: with a 3-bit floor it quantizes 96.4% of the weights to 4.5 bits per parameter at 6.933 perplexity, within 4.4% of full precision (6.643) at 3.56x compression, against 7.158 for JAB at the same budget. Which matrix a weight sits in matters more than any sensitivity estimate we computed. Two things came out sideways. Block-local reconstruction is an unreliable proxy for end-to-end perplexity: one run improved a block's own objective 4.6x while perplexity rose 32x, which is why every allocation here is validated end-to-end. And on attention-only quantization, fine-tuning moved weights farther from their pretrained values while pulling attention outputs closer, with net gains. Post-training seems to recover attention behavior, not weights.

1 INTRODUCTION

JAB tests whether one attention-output objective can both refine quantized weights and allocate bits, addressing the proxy gap between reconstruction and sensitivity criteria. The reported results distinguish attention-only success from weaker transfer to MLP and full-model allocation.

Motivation

  • JAB addresses the proxy gap: most PTQ pipelines use a reconstruction objective different from the criterion that determines bit widths.The paper asks whether one objective can both produce quantized weights and set their budget.Source: §1 · p. 2

Method

  • JAB defines a joint QKV attention loss and uses it to refine quantized weights and score blocks for multiple-choice knapsack allocation.It starts from GPTQ weights, then uses straight-through fine-tuning with learnable scales; Hessian-trace probes provide the allocation score.Source: §1 · p. 2

Findings

  • At 3 bits, JAB recovers 77–90% of the uniform-GPTQ-to-full-precision gap across four Mistral-7B corpus pairings, but its advantage does not transfer to MLP or full-model allocation.A role-offset rule beats JAB on the full model; with a 3-bit floor it reaches 6.933 perplexity at 4.5 bits per parameter, versus 7.158 for JAB at the same budget.Source: §1 · p. 2

Contribution

  • Treating Q, K, and V separately agrees with the joint loss in expectation but can underestimate its variability.The paper distinguishes the joint objective from per-matrix treatment because cross-projection interactions affect the spread of the loss.Source: §1 · p. 2

Pipeline

  • The pipeline separates bit allocation from quantization and refinement, while other quantizer improvements are held fixed as orthogonal to allocation.GPTQ supplies initial weights before the joint attention loss is minimized through the quantizer.Source: Figure 1 · p. 3, §1 · p. 3

Contribution

  • Layer role can provide a stronger allocation signal than curvature outside attention, while better block-local reconstruction does not guarantee better end-to-end perplexity.On attention-only quantization, refinement can also move weights farther from their floating-point values while making attention outputs more accurate.Source: §1 · p. 3

2 METHOD

The method uses GPTQ as a baseline, then defines a joint attention loss for QKV refinement and allocation. It estimates sensitivity from that loss, allocates bits under a budget, and evaluates both weight- and attention-space fidelity.

2.1 LAYER-WISE PTQ

  • GPTQ quantizes a fused QKV projection using a local weight-reconstruction surrogate, with block Hessians accumulated in the model's current quantized state.The solve uses float64 because the float32 column loop can make perplexity non-monotone with bit width.Source: §2.1 · p. 3

2.2 THE JOINT ATTENTION LOSS

  • The joint attention loss compares quantized and unmodified attention outputs and post-softmax maps for a block.The effective KL-to-MSE ratio is 10^3–10^4 at λKL = 0.1, making the optimized loss primarily attention-map distillation with output-MSE regularization.Source: §2.2 · pp. 3–4

2.3 WHY ONE JOINT OBJECTIVE RATHER THAN THREE

  • Because attention scores depend jointly on WQ and WK, separate matrix treatment discards nonzero cross-block Hessian terms and can misrepresent perturbation risk.The joint Hessian has a null space where the separable model reports damage despite none; the models agree in expectation but differ in spread.Source: §2.3 · p. 4

2.4 SENSITIVITY AND ALLOCATION

  • Hutchinson probes estimate the Hessian trace of the same joint loss, and an MCKP allocator chooses each layer's bit width from candidate scores under the budget.The oracle instead measures end-to-end logit KL by quantizing each layer separately, requiring approximately 73 model loads.Source: §2.4 · p. 4

2.5 JOINT FINE-TUNING THROUGH THE QUANTIZER

  • Fine-tuning starts from GPTQ's quantized solution and descends the joint loss through the quantizer using a straight-through estimator and learned group scales.Held-out batches gate write-back after 200 Adam steps.Source: §2.5 · p. 4

2.6 METRICS

  • Evaluation reports perplexity, next-token accuracy, weight-space error, attention-space error, and their amplification ratio.Table 1 covers attention-only Mistral-7B at 3 bits across four calibration-to-evaluation pairings and reports the share of the uniform-to-fp16 gap closed.Source: §2.6 · p. 4, Table 1 · p. 5

3 EXPERIMENTAL SETUP

The experiments cover GPT-2 small and Mistral-7B, with attention-only, MLP-only, and full-model quantization studies.

Experimental scope

  • Experiments use GPT-2 small and Mistral-7B, with separate attention-only, GPT-2 MLP-only, and Mistral-7B full-model studies.Results are single-seed; Mistral perplexities use 32 windows of 512 tokens and are comparable within a study, not to published full-split figures.Source: §3 · p. 5

4 ATTENTION-ONLY QUANTIZATION

Attention-only allocation works well at 3 bits on Mistral-7B, but mixed precision can fail at an exact 3-bit budget on GPT-2. Fine-tuning improves attention behavior while moving weights farther from their floating-point values.

Attention-only allocation

  • On Mistral-7B, adaptive allocation beats uniform GPTQ at 3 bits across four corpus pairings, recovering 77–90% of the uniform-versus-fp16 gap and 2–3 accuracy points.By 4 bits, all methods are within 1–2% of fp16; at 3 bits allocation is worth 3–5 perplexity points, while fine-tuning changes perplexity by at most 0.17 in either direction.Source: §4 · p. 5

Sensitivity and allocation

  • Across four GPT-2 budgets of at least 3.5 bits, the Hutchinson criterion and an approximately 73-load oracle differ by 0.17 perplexity on average.On Mistral at 3 bits, second-order scoring beats the first-order Fisher proxy, while greedy and Pareto solvers are indistinguishable.Source: §4 · p. 5

Mixed-precision boundary

  • At exactly 3.0 bits on GPT-2, adaptive allocation reaches 86.36 perplexity against uniform's 38.35 because at least one unit must fall to 2 bits.The additive objective misses the low-bit cliff; depth re-weighting halves the damage, while mixed precision performs better at fractional budgets.Source: §4 · p. 5

Refinement

  • At 4 bits, fine-tuning raises weight error by 19% while lowering attention error by 33%, and both perplexity and accuracy improve.Across the reported configurations, the KL term supplies 88% of the gain; MSE-only refinement lowers attention error but still degrades perplexity.Source: §4 · p. 5

5 BEYOND ATTENTION: THE MLP

The GPT-2 MLP experiment tests which parts of JAB transfer when its attention-specific signal is removed. The Hessian criterion fails to identify useful allocation structure, which the oracle assigns by matrix role rather than depth.

5 BEYOND ATTENTION: THE MLP

  • The GPT-2 experiment quantizes mlp.c fc and mlp.c proj with attention kept in fp32, using execution-order Hessians and output MSE without the attention-map KL term.The MLP contains 66.6% of non-embedding weights, versus 25.0% for QKV.Source: §5 · p. 6
  • 120.70 vs. 53.71 perplexity at 3 bits and 27.46 vs. 26.23 at 4 bits: Hessian allocation performs worse than uniform MLP quantization.At 3 bits, the oracle reaches 46.15 perplexity, a 14% reduction versus uniform.Source: §5 · p. 6
  • At every fractional budget, the oracle assigns mlp.c fc to b+1 and mlp.c proj to b−1 in every block, splitting by matrix role rather than depth.The concatenated block trace is identical for both units, so the criterion's perturbation factor is the only signal that could distinguish them.Source: §5 · p. 6

6 FULL-MODEL QUANTIZATION

Full-model experiments find that role-based allocation outperforms curvature-based allocation at matched budgets, while a 3-bit floor avoids the catastrophic low-bit regime. They also show that improvements to block-local objectives can coincide with much worse end-to-end perplexity.

6 FULL-MODEL QUANTIZATION

  • Full-model experiments allocate one shared budget first across GPT-2’s fused QKV and MLP matrices, then across all linear modules in Mistral-7B.The Mistral-7B allocation covers 224 units and 96.4% of model parameters.Source: §6 · p. 6, §6.1 · p. 6, §6.2 · p. 7

6.1 GPT-2: QKV AND MLP TOGETHER

  • At fractional budgets, Type-offset reaches 64.896 at 3.5 bits and 28.211 at 4.5, versus JAB+prop’s 67.337 and 28.694; only near saturation at 6 bits does uniform win.JAB’s attention-defined sensitivity over-invests in QKV and under-funds the MLP, which dominates the parameter count.Source: §6.1 · p. 6, Table 3 · p. 7

6.2 MISTRAL-7B: ALL SEVEN MODULES TOGETHER

  • The Mistral-7B study budgets all seven module types by parameter count because the MLP holds 80.8% of weights and module sizes span 14×.Type-offset assigns fixed integer offsets by module kind, then makes single-unit moves to meet the budget.Source: §6.2 · p. 7

6.3 LAYER ROLE OUTPERFORMS CURVATURE

  • At B = 4.5 bits per parameter, Type-offset reaches 6.970 perplexity against 7.250 for JAB and 10.770 for JAB-norm, with all three at the same budget.Type-offset uses seven integer offsets and no Hessian or Hutchinson probes.Source: §6.3 · p. 7, §6.2 · p. 7

6.4 BLOCK-LOCAL OBJECTIVES CAN DIVERGE FROM END-TO-END QUALITY

  • Block 0’s local objective improves 4.6×, yet end-to-end perplexity rises from 7.373 to 239.52 after refinement.Under uniform 4-bit quantization, the same refinement stage raises perplexity 1.6× rather than 32.5×; the authors warn that locally validated objectives do not guarantee global quality.Source: §6.4 · p. 8

6.5 A 3-BIT FLOOR WITH PER-UNIT CURVATURE

  • At B = 4.5, Type-offset reaches 6.933 perplexity versus 7.158 for JAB, while the 3-bit floor avoids the catastrophic regime and yields 3.56× compression.The result is 4.4% higher perplexity than fp16; the quantized weights use 3.93 GB.Source: Table 4 · p. 7, §6.5 · p. 9

7 CONCLUSION

JAB uses one loss for attention reconstruction and allocation, succeeding on attention but not transferring to MLP or full-model allocation. The conclusion argues that matrix role can be a stronger signal than curvature, while warning that local improvements need not improve end-to-end quality.

7 CONCLUSION

  • On attention, adaptive allocation recovers most of the uniform-to-fp16 gap at 7B, but does not transfer beyond attention; matrix role can carry more signal than the curvature computed.Per-unit resolution does not change that result, since the curvature comes from an attention objective.Source: §7 · p. 9
  • At 3.5 bits, curvature beats the role prior (7.51 against 7.87), while the paper tests only models up to 7B.The conclusion notes that the lower-bit regime may grow in importance with model size, but this remains untested beyond 7B.Source: §7 · p. 9
  • The authors report using generative AI for manuscript editing, LaTeX, code, and number-checking, but not to produce experimental results.They state that all reported numbers were computed from model outputs and that AI-drafted scripts were run and checked.Source: AI USE STATEMENT · p. 9

REPRODUCIBILITY STATEMENT

The supplementary material documents implementation and evaluation choices, positions JAB against prior methods, and develops the argument for a joint QKV objective. It also details fine-tuning, sensitivity scoring, allocation heuristics, and metrics.

REPRODUCIBILITY STATEMENT / A POSITIONING AGAINST PRIOR WORK

B.1 GPTQ CORE

  • GPTQ rounds weights using inverse-Hessian compensation, with group-wise scaling and activation ordering; the Mistral solve uses float32 after float64 Hessian accumulation because float64 solving is 32× slower.The authors flag precision as consequential: fp32 rounding can produce non-monotone perplexity across bit-widths.Source: §B.1 · p. 12

B.2 EFFECTIVE WEIGHT OF THE KL TERM

  • The implemented KL term dominates output MSE by a measured ratio of 10^3–10^4 at λKL = 0.1, so optimization acts mainly as attention-map distillation.The authors retain this convention and quantify it by ablation.Source: §B.2 · p. 12

B.3 WHY ONE JOINT OBJECTIVE RATHER THAN THREE: FULL ARGUMENT

  • Separate Q, K, and V treatment discards structurally nonzero cross-block curvature: its estimate can miss true joint damage, despite being unbiased on average, and is over-confident in the tail.The argument applies to reconstruction and fine-tuning; for scoring, the traces of joint and separate Hessians coincide.Source: §B.3 · p. 13

B.4 FISHER-COUPLING

  • Fisher-coupling scores a layer by the squared gradient norm at a coarse 2-bit probe, providing a cheaper first-order proxy rather than a decomposition of the joint Hessian.It is compared with second-order scoring in Section D.Source: §B.4 · p. 13

B.5 ALLOCATORS

  • The greedy and MCKP/Pareto heuristics consume the same sensitivity table but use different downgrade-and-upgrade procedures; the latter first restricts choices to Pareto-efficient points.MCKP is NP-hard, so exhaustive search is described as intractable.Source: §B.5 · p. 14

B.6 JOINT STE FINE-TUNING: THE FIVE DESIGN DECISIONS

  • Joint STE fine-tuning uses straight-through gradients for weights and LSQ gradients for scales, while matching teacher and student inputs and gating checkpoint write-back on held-out performance.Expressing the learning rate in grid-step units raises flip rate from 0.000 in eleven of twelve blocks to 0.12–0.18.Source: §B.6 · p. 14

B.7 EVALUATION METRICS IN FULL

  • The evaluation reports perplexity, next-token top-1 accuracy, weight-space error, and attention-space error; ρ = εA/εW indicates whether attention amplifies or attenuates relative weight perturbation.The attention-space teacher and student are evaluated on the same input from the live partially quantized model.Source: §B.7 · pp. 14–15

C EXPERIMENTAL SETUP IN FULL

The experiments cover QKV quantization in GPT-2 and Mistral-7B across calibration and evaluation settings. Sensitivity varies by layer and architecture, but the bit-width cliff is shared, and rankings are stable across calibration corpora.

C EXPERIMENTAL SETUP IN FULL

  • The setup uses group size 128, activation ordering, λKL = 0.1 unless ablated, and 10 Hutchinson probes on two calibration batches per layer.Mistral runs use a layer-streaming re-implementation and 32 windows per configuration, so those perplexities are subset figures rather than comparable published results.Source: C EXPERIMENTAL SETUP IN FULL · p. 15

D THE SENSITIVITY CRITERION

  • Mistral-7B QKV allocation recovers 77–90% of the uniform-versus-fp16 perplexity gap across four corpus pairings, and every method is within 1–2% of fp16 by 4 bits.The figure plots normalized perplexity and the share of the gap recovered at 3 bits.Source: Figure 3 · p. 16
  • Sensitivity spans 41× across GPT-2 layers and 135× across Mistral layers, but the depth profile differs: GPT-2 peaks early, while Mistral peaks in its final layers.The models are GPT-2 small and Mistral-7Bv0.1; only QKV is quantized, so their absolute perplexities are not comparable to end-to-end quantized systems.Source: C EXPERIMENTAL SETUP IN FULL · p. 15, D THE SENSITIVITY CRITERION · p. 16
  • Dropping from 4 to 3 bits multiplies the sensitivity score by roughly 5×, and dropping from 3 to 2 adds another 5–7×, nearly independently of layer.The shared cliff leaves little layer differentiation until the budget forces a sub-4-bit choice.Source: D THE SENSITIVITY CRITERION · p. 16
  • Traces calibrated on WikiText-2 and C4 correlate at r = 0.9995 and induce nearly the same layer ordering.Only one adjacent pair differs, by under 1%.Source: D THE SENSITIVITY CRITERION · p. 16
  • At 3 bits, the second-order Hutchinson score beats the first-order Fisher proxy on Mistral-7B (5.268 vs. 5.302 perplexity), while greedy and MCKP/Pareto are indistinguishable given identical scores.At 6 bits, all four combinations lie within 0.007 perplexity.Source: D THE SENSITIVITY CRITERION · p. 16

E BIT ALLOCATION

Bit allocation helps at some GPT-2 budgets but can fail catastrophically below 4 bits; Mistral-7B instead shows strong gains from adaptive QKV allocation at 3 bits. The authors also find that a costly oracle adds little at most tested budgets.

E BIT ALLOCATION

  • At 3.0 bits, GPT-2 adaptive allocation reaches 86.36 perplexity against 38.35 for uniform 3-bit, while at 4.0 bits it reproduces the uniform baseline exactly.At 6 average bits, adaptive allocation improves on uniform 4-bit by −1.18 perplexity.Source: E BIT ALLOCATION · p. 16, Table 8 · p. 17
  • The 3-bit GPT-2 allocation must contain at least one 2-bit block, and the additive score misses the severe low-bit perplexity risk; depth re-weighting reduces the damage to 43.93 from 86.36.The derivation uses the 3-bit floor and budget arithmetic to show that any non-uniform assignment must include a sub-3-bit block.Source: E BIT ALLOCATION · pp. 16–17
  • At 2.5 and 3.5 average bits, mixed precision beats geometric interpolation between neighboring uniform solutions by 2.2× and 10.7%, respectively.Source: E BIT ALLOCATION · p. 17
  • Across budgets of at least 3.5 bits, a roughly 73× more costly oracle differs from the cheap criterion by a mean absolute 0.17 perplexity, and the cheap criterion leads in half the cases.The oracle helps decisively only at 3.0 bits, where neither it nor depth re-weighting works as well as avoiding mixed precision.Source: E BIT ALLOCATION · p. 17
  • At 3 bits, Mistral-7B adaptive allocation recovers 77–90% of the uniform-versus-fp16 perplexity gap across all four corpus pairings.By 4 bits, every method is within 1–2% of fp16; the authors note the 3-bit collapse seen on GPT-2 does not appear in Mistral.Source: E BIT ALLOCATION · p. 17, Table 9 · p. 17
  • Adding joint STE fine-tuning gives large relative perplexity gains where quantization damage is large, but gains shrink and become model- and corpus-dependent in the usable regime.On GPT-2, removing the KL term is harmful at every budget on both corpora and catastrophic below 4 bits; on Mistral its effect stays within ±1%.Source: Figure 4 · p. 18

F QUANTIZATION METHOD: GPTQ VS. JOINT FINE-TUNING

Joint fine-tuning often improves GPT-2 quantization, but held-out local-loss gains can coexist with perplexity regressions. The KL attention-map term supplies most of the gain, while weight movement alone does not indicate benefit.

F QUANTIZATION METHOD: GPTQ VS. JOINT FINE-TUNING

  • Fine-tuning improved 18 of 21 GPT-2 configurations, with gains of 1–2% in the usable regime and the best arm at 24.786 perplexity against 24.357 for fp32.Gains scaled to −45% to −90% below 3 bits; flip rates of 0.12–0.18 indicate changes to deployed weights.Source: F QUANTIZATION METHOD: GPTQ VS. JOINT… · p. 18
  • At 3.5 average bits, three runs improved held-out attention loss while perplexity worsened slightly, and the sign reversed on C4 evaluation.The same weights scored on both corpora isolate evaluation-corpus effects; both observed instability patterns were confined to fractional or aggressive budgets.Source: F QUANTIZATION METHOD: GPTQ VS. JOINT… · p. 18
  • Removing the KL term retained only 0.052 of the total 0.444 fine-tuning gain, while MSE-only lowered attention-output error but worsened perplexity.The KL term supplied 88% of the gain; at 4 bits MSE-only reduced εA from 0.238 to 0.188.Source: F QUANTIZATION METHOD: GPTQ VS. JOINT… · pp. 18–19
  • Across budgets and both evaluation corpora, the amplification-factor ordering was GPTQ-only > MSE-only > MSE+KL, without a crossing.Figure 5 also plots attention-space against weight-space error and normalized accuracy against perplexity.Source: Figure 5 · p. 19
  • At fixed bit widths, MSE-only moved more weights to new grid positions than KL fine-tuning (0.177 vs. 0.136) yet ended 0.392 perplexity worse.Benefit increased with fine-tuning depth; limiting the stage to four blocks landed between GPTQ-only and full-depth results.Source: F QUANTIZATION METHOD: GPTQ VS. JOINT… · p. 19

G WHERE THE ERROR LIVES, AND WHETHER ACCURACY AGREES

Fine-tuning can move quantized weights farther from their pretrained values while bringing attention outputs closer, with task metrics improving. Allocation transfers across corpus mismatch, whereas refinement is more corpus-sensitive.

G WHERE THE ERROR LIVES, AND WHETHER ACCURACY AGREES

  • At 4 bits, fine-tuning moved weights 19% farther from float (εW 0.174 → 0.208) while moving attention output 33% closer (εA 0.238 → 0.160), with both perplexity and accuracy improving.The same trade appeared at 3 and 6 bits.Source: G WHERE THE ERROR LIVES, AND WHETHER… · p. 20
  • On Mistral at 3 bits, adaptive allocation reduced attention-reconstruction error versus uniform while increasing weight error under both calibration corpora.With WikiText-2 calibration, εA was 1.13 vs. 1.53 × 10−3 and εW was 0.467 vs. 0.433; with C4 calibration, εA was 1.32 vs. 1.65 × 10−3 and εW was 0.474 vs. 0.439.Source: G WHERE THE ERROR LIVES, AND WHETHER… · p. 20
  • Next-token accuracy was nearly rank-inverse to perplexity across GPT-2 arms (Spearman ρs = −0.996 on WikiText-2 and −0.990 on C4).The relation also held across models, corpora, and modes after normalization by each configuration’s full-precision baseline.Source: G WHERE THE ERROR LIVES, AND WHETHER… · p. 20

H CROSS-CORPUS ROBUSTNESS

  • Uniform 4-bit GPTQ incurred +8.9% perplexity against the WikiText-2 control and +9.3% against the C4 control despite differing baselines.The absolute baselines differed by 6.4 points.Source: H CROSS-CORPUS ROBUSTNESS · p. 20
  • Under corpus mismatch, Mistral adaptive allocation recovered 81.4% and 85.0% of the uniform gap, compared with 89.5% and 76.6% for matched pairings.Allocation did not change sign under either intervention; the reported fine-tuning instabilities were corpus-induced.Source: H CROSS-CORPUS ROBUSTNESS · p. 20

I BEYOND QKV: THE MLP SUBLAYER

The MLP and full-model experiments show that attention-derived curvature scores do not reliably capture matrix-role allocation, while block-local refinement can damage end-to-end quality. In the full Mistral-7B setting, a simple role-based offset rule outperforms JAB at a matched 4.5-bit budget.

I BEYOND QKV: THE MLP SUBLAYER

  • On GPT-2’s MLP, the Hessian criterion lost to uniform at both comparable budgets, while the oracle improved on uniform at 3 bits by 14%.The oracle’s fractional-budget assignment split bits by matrix role without depth variation; the criterion instead followed a depth gradient.Source: I BEYOND QKV: THE MLP SUBLAYER · p. 21, Table 12 · p. 22
  • The GPT-2 MLP trace rose from block 5 to the final layer and spanned only 4.5×, unlike the QKV trace’s 41× spread and earlier peak.The MLP trace therefore inverts the QKV depth profile within the same model.Source: I BEYOND QKV: THE MLP SUBLAYER · p. 21
  • The MLP trace is identical for both matrices in a block, so scoring cannot distinguish them despite their different input dimensions, distributions, and oracle sensitivities.The pooled trace leaves discrimination between the two allocation units to the perturbation factor.Source: I BEYOND QKV: THE MLP SUBLAYER · p. 21
  • In the MLP sweep, fine-tuning made no weight changes in the deepest blocks across many arms, and the paper treats its Objective-3 results as unresolved.The authors attribute the zero-flip signature to a fixed absolute learning rate; the objective was unchanged to six decimals in those blocks.Source: I BEYOND QKV: THE MLP SUBLAYER · p. 22

J FULL-MODEL QUANTIZATION: EXTENDED ANALYSIS

  • In the full-model Mistral-7B experiment, every linear module in every decoder layer was quantized under a single budget, covering 6.98B parameters (96.4% of the model), across 224 allocation units.Source: J FULL-MODEL QUANTIZATION: EXTENDED… · p. 23

J.2 LAYER ROLE OUTPERFORMS CURVATURE

  • At exactly 4.500 bits per parameter on Mistral-7B, Type-offset reached 6.970 perplexity versus 7.250 for JAB and 10.77 for JAB-norm, without Hessian scoring.The comparison is the second architecture in which role-based allocation matches or beats the learned criterion; Figure 6 also shows Type-offset best at B = 4.5.Source: §J.2 · p. 23, Figure 6 · p. 24

J.3 BLOCK-LOCAL OBJECTIVES CAN DIVERGE FROM END-TO-END QUALITY

  • Refining one Mistral block lowered its local objective 4.6× yet raised perplexity from 7.373 to 239.52; the change was confined to block 0.Block 0 was assigned 2 bits on six of seven units and feeds the remaining 31 blocks; uniform 4-bit refinement degraded perplexity by 1.6× instead.Source: §J.3 · p. 25

J.5 GPT-2: QKV+MLP, FULL WIKITEXT-2 TABLE

  • With GPT-2 QKV and MLP sharing a budget, Type-offset led at 3.5 and 4.5 bits (64.896 and 28.211), while uniform won at 6 bits (24.546).Attention error was uninformative in this combined setting: Type-offset had the worst εA at 3.5 and 4.5 bits despite best perplexity.Source: §J.5 · p. 26

K LIMITATIONS IN FULL

The limitations include single-draw or single-seed evidence, incomplete coverage across models and modules, and unresolved estimator, solver, and optimizer questions.

K LIMITATIONS IN FULL

  • The reported trace estimates use ten Hutchinson probes on two batches, without repeated-seed variance, so key trace gaps are not shown to exceed sampling noise.The cited examples include the block-0/block-1 gap behind the 3-bit failure and the 4.5× MLP depth gradient.Source: K LIMITATIONS IN FULL · p. 28
  • Full-model composition was tested on Mistral only, leaving QKV–MLP interactions on GPT-2 unmeasured; embeddings and the output head are never quantized.The scope statement caps GPT-2 end-to-end compression at 1.66× at 4 bits.Source: K LIMITATIONS IN FULL · p. 28
  • The full-scale greedy allocator’s optimality gap was not measured, so the 3-bit failure cannot be apportioned between criterion and solver.Agreement between greedy and MCKP/Pareto on Mistral is described as suggestive, not proof.Source: K LIMITATIONS IN FULL · p. 28
  • No Q/K/V-separate arm was run, leaving the real-block effect of the predicted joint-versus-separate loss gap unknown.The cross-block terms are structurally non-zero, but their magnitude relative to the Hessian was not estimated.Source: K LIMITATIONS IN FULL · p. 28
  • The weak MLP fine-tuning result is confounded by zero flips in deep blocks; a pooled grid-step calculation may be an optimizer defect.The report says the result cannot be separated from that defect until the step is confirmed to be computed per unit.Source: K LIMITATIONS IN FULL · p. 28
  • The Mistral results are one-seed evidence, and the implementation was not checked against the full GPT-2 validation suite.The limitation passage also notes a float32 rather than float64 solve and a different εW, εA normalization.Source: K LIMITATIONS IN FULL · p. 28

Related papers

Loading 2609.34866v1…