PaperlayerView exact revision on arXiv

Source-linked AI summary

Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit

Zexin Zhuang, Yanhang Li, Zhichao Fan

arXiv:2605.28873v1cs.LG

TL;DR

Quantization studies need an ex-ante way to distinguish small effects from effects that the evaluation protocol cannot resolve. This paper adapts paired-binary planning into a conservative MDE budget and audits FP16 versus NF4 across benchmark splits and MMLU prompt templates. The audit finds that most split variance is binomial sampling noise and that prompt-template variation can match or exceed observed quantization deltas.

  • Problem

    Evaluation reports often do not establish the smallest quantization effect their protocol could reliably detect, conflating small effects with limited detectability.

  • Method

    The paper derives a conservative paired MDE bound and applies it to an audit of four models, four benchmarks, five splits, and an MMLU prompt-template study.

  • Results

    25 of 32 cross-split SDs fall within ±1.5 pp of the binomial reference, while MMLU prompt-template variance of 2–10 pp meets or exceeds the largest observed quantization delta of 3.2 pp.

  • Takeaways & Limitations

    Benchmark designers should preregister an MDE and fix prompt templates before interpreting precision comparisons.

  • Takeaways & Limitations

    The audit does not retain per-example correctness, uses an unmeasured planning value for ρ_d, tests only NF4, and has only two 7B models.

Abstract

from arXiv · show

This is a planning-method note with an unpaired pilot audit. We adapt the classical paired-binary sample-size calculation (Miettinen, 1968) to quantization benchmarks, giving a conservative minimum detectable effect (MDE) bound $δ^{*} \le (z_{1-α/2}+z_{1-β})\sqrt{ρ_d/m}$ in the paired item count $m$ and the FP16-NF4 disagreement rate $ρ_d$. The bound turns "how reliable is my quantization claim?" into a one-line budget a benchmark designer can commit to before running. We illustrate the bound on four models and four benchmarks ($k=5$ splits of $n=100$), and add a parallel MMLU prompt-template study to put the bound's quantization-noise scale alongside the prompt-noise scale. Assuming $ρ_d=0.10$ (an unmeasured planning value), all observed NF4-FP16 deltas fall below the implied MDE, and most cross-split SDs lie within $\pm 1.5$ pp of the binomial reference $\sqrt{p(1-p)/n}$, so much of the variance reported as "benchmark unreliability" on $n=100$ subsamples is binomial sampling noise. The single borderline cell (OPT-WinoGrande, $|Δ|=3.2$ pp) is below the implied MDE at $ρ_d=0.10$ but above it at $ρ_d=0.05$, illustrating the planning trade-off the bound makes explicit. On MMLU, prompt-template ranges of 2-10 pp meet or exceed the largest observed quantization delta (3.2 pp), so a quantization audit that does not first fix the prompt template absorbs template variance into its noise floor. We complement the bound with a five-line pre-registration template.

1. Introduction

The paper reframes quantization reliability as a minimum-detectable-effect problem, distinguishing small population effects from effects the evaluation protocol cannot resolve. It proposes an ex-ante paired MDE budget, audits four models and four benchmarks, and uses QRI as a descriptive noise diagnostic.

  • Motivation: The central question is the smallest quantization effect that an evaluation protocol could reliably detect, given benchmark size and baseline accuracy.
  • Motivation: Reports of small quantization effects can conflate a genuinely small population effect with a protocol that cannot detect small effects.
  • Contributions: The paired MDE bound uses paired item count m, disagreement rate ρ_d, significance α, and power 1−β to create an ex-ante quantization-audit budget.
  • Contributions: The pilot evaluates four models and four benchmarks using k=5 non-overlapping splits of n=100 items, while separately measuring MMLU prompt-template sensitivity.
  • Contributions: QRI is a descriptive signal-to-noise diagnostic, not a hypothesis test; power-controlled decisions compare |∆̂| directly with δ*.
  • Findings: 25 of 32 observed cross-split SDs fall within ±1.5 pp of the binomial reference, indicating that much reported unreliability is small-n binomial sampling noise.
  • Recommendations: The paper recommends preregistering an MDE target, reporting paired discordant counts, separating sampling from subset-composition noise, and sweeping prompt templates.

2. Related Work

The paper positions its contribution within post-training quantization, benchmark-evaluation, and prompt-sensitivity research, then formalizes a conservative paired MDE framework for quantization audits. It distinguishes descriptive QRI from power-controlled decisions and specifies how paired sample size, split structure, and unmeasured disagreement affect interpretation.

  • Related work: Prior work covers GPTQ, NF4, AWQ, SmoothQuant, SpQR, and ZeroQuant, while benchmark studies address evaluation breadth, subset selection, and statistical fragility.
  • Related work: Prompt-sensitivity studies motivate sweeping templates; this paper examines quantization–prompt interaction only at the n=50 MMLU scale and treats it as exploratory.
  • Paired framework: The paired framework defines FP16 and NF4 correctness indicators, per-item differences D_i, disagreement rate ρ_d, and population effect δ under iid, deterministic-output, and preregistered-disagreement assumptions.
  • Paired framework: The MDE bound is conservative because it uses ρ_d as a variance proxy and a two-sided z-test, yielding a preregisterable sufficient bound rather than an impossibility threshold.
  • Sample-size targets: Equation 2 applies to the paired sample used for the estimand: m=n for a single split or m=kn for an aggregate across k non-overlapping splits.
  • Noise interpretation: Cross-split SD has binomial scale under independent sampling, so comparing aggregate quantization effects with split-level noise mixes scales and should be avoided.
  • Limitations: The audit cannot retrospectively measure ρ_d because per-example correctness was not retained, so empirical deltas are treated as observations rather than tested differences.

4. Methods

The methods evaluate FP16 and NF4 across four language models and four benchmarks using repeated non-overlapping subsamples, with a separate MMLU prompt-template sweep. Accuracy is scored with fixed task-specific procedures, and results are reported as observed means and cross-split standard deviations.

  • Models: The study evaluates OPT-2.7B, Pythia-2.8B, Llama-2-7B, and Mistral-7B at FP16 and NF4 precision.
  • Quantization: NF4 is applied through BitsAndBytes with 4-bit NF4 quantization, float16 compute, and disabled double quantization.
  • Benchmarks and splits: Each benchmark is randomly partitioned into k=5 non-overlapping splits of n=100 examples using seed 42.
  • Reporting: Table 1 reports benchmark accuracy as mean±cross-split SD on the observed subsamples, while population confidence intervals are wider.
  • Scoring: Multiple-choice tasks use answer-only continuation likelihoods under a fixed prompt template, while WinoGrande uses likelihood ratios for its two pronoun resolutions.
  • Prompt study: MMLU prompt sensitivity is measured with three templates on n=50 examples per template and is not extended to ARC, WinoGrande, or HellaSwag.

5. Results: Illustrative Pilot Audit

The pilot audit illustrates how binomial sampling, quantization effects, and prompt-template variation shape the detectability of NF4–FP16 differences. Most cross-split variation tracked the binomial reference, while OPT-WinoGrande remained the sole borderline delta under alternative disagreement assumptions.

  • Study scope: The pilot did not retain per-example correctness, so its results cannot estimate ρd, run paired McNemar tests, or validate the paired audit machinery.The numerical findings are illustrative applications of the proposed methodology rather than tested population claims about NF4.
  • Quantization deltas: At m=500, the aggregate paired MDE was approximately 4.0 pp for ρd=0.10 and 2.8 pp for ρd=0.05.These planning values were evaluated at α=0.05 and power 0.80.
  • Observed cross-split variation: 25 of 32 cells fell within ±1.5 pp of the binomial reference, indicating that much observed cross-split variation on n=100 subsamples was binomial sampling noise.The largest positive residuals concentrated on MMLU and 3B-tier ARC-Easy, with MMLU residuals linked to subject-mix variation.
  • Prompt sensitivity: MMLU template ranges of 2–10 pp were comparable to or larger than the largest observed quantization delta of 3.2 pp.The audit therefore treats prompt-template variation as part of the precision-comparison noise floor and recommends fixing the template before comparison.
  • Quantization Reliability Index: QRI_split ranged from 0.09 to 0.76, while QRI_combined ranged from 0.07 to 0.18 across the four MMLU cells.QRI is presented as a descriptive signal-to-noise heuristic, whereas power-controlled decisions use direct comparison of |∆̂| with δ*.
  • Quantization deltas: The largest 3B accuracy delta was −3.2 pp for OPT WinoGrande, below the ρd=0.10 MDE but above the ρd=0.05 MDE.Its detectability therefore depends on the unmeasured disagreement-rate planning value.

6. Discussion

The audit finds that observed quantization differences are small relative to binomial sampling noise and the paired MDE, while prompt-template variation can be as large as the quantization signal. The discussion therefore distinguishes detectability limits from population effects and calls for ex-ante sample-size planning.

  • Detectability: FP16-vs-NF4 accuracy differences were small relative to both binomial sampling SD and the paired MDE, but n=100 cannot distinguish a small population effect from insufficient detectability.The audit therefore cannot resolve whether the population effect is genuinely small or merely below the protocol’s detection threshold.
  • Cross-split variance: 25 of 32 cells had cross-split SD within ±1.5 pp of the binomial reference, making small-sample noise a more plausible explanation than an NF4-specific effect.The largest residual was +3.0 pp on OPT-NF4 MMLU, consistent with subject-mix variance from random partitions of MMLU’s 57 subjects.
  • Prompt variance: MMLU prompt-template ranges of 2–10 pp were comparable to or larger than the largest observed quantization deltas, so single-template comparisons absorb template variance.The discussion recommends reporting prompt-template variance for every benchmark used in a quantization claim.
  • Scope of the guarantee: The MDE bound guarantees detectability, not expected effect magnitude, leaving prediction of ρd and δ from calibration data or model statistics as future work.The paper describes this predictor as the missing bridge between theory and benchmark planning.

7. Recommendations

The recommendations turn quantization evaluation into an auditable protocol: preregister detectability, retain paired data, decompose sampling variance, measure prompt variance, and avoid overgeneralizing from narrow model or method coverage.

  • Pre-registered MDE: Preregister α, 1 −β, ρd, m, and δ∗, and label effects below δ∗ as not power-distinguishable rather than small.For k non-overlapping splits of n items, state whether the paired count is m=n or m=kn.
  • Paired statistics: Retain per-example correctness so paired McNemar or bootstrap estimators can use the paired covariance that single-split accuracy SDs omit.Positive FP16–NF4 correctness correlation typically makes paired-delta noise smaller than unpaired accuracy noise.
  • Binomial-reference decomposition: Report cross-split SD alongside the binomial reference σbin(p̂,n) and classify a small gap as subset-composition variance rather than population noise.A large gap is instead treated as an estimate of population noise.
  • Prompt variance: Pre-register at least three prompt templates, pair them across precisions, and report template variance and QRIcombined on every benchmark.The recommendation extends the MMLU prompt sweep beyond the benchmark where prompt variance was measured in the pilot.
  • Coverage: Use multiple models, families, and quantization methods because n=1 or n=2 models per scale tier cannot support broad generalization.Cross-method comparison includes NF4, GPTQ, AWQ, and SpQR to control for method-specific quirks.

8. Pre-Registration Template

The five-line template makes detectability commitments explicit before evaluation by fixing the estimand, test parameters, disagreement prior, computed MDE, and revision rule. It gives reviewers a pre-numeric check on which claims the protocol could license.

  • Purpose: The completed template is an ex-ante guarantee on detectability that lets reviewers check which claims the protocol could license before benchmark numbers appear.
  • Estimand: The template first states whether the estimand is single-split m=n or aggregate m=kn, including k and n.The pilot uses m=500 aggregate.
  • Disagreement prior: The disagreement-prior line records ρprior_d with justification from a calibration set, prior literature, or a sensitivity range.
  • Computed MDE: The computed-MDE line evaluates δ∗(m, ρprior_d, α, 1 −β) from Eq. 2 and reports it in percentage points.
  • Revision rule: The revision rule retains per-example correctness, reports U95(ρd), and re-evaluates borderline claims under a larger MDE when U95(ρd) exceeds ρprior_d.Otherwise, the preregistered MDE remains binding.

9. Limitations

The paper’s limitations constrain inference to an unpaired NF4 pilot with narrow model and prompt coverage, and qualify the MDE as a conservative normal-approximation bound under assumptions that MMLU subject clustering can violate.

  • Paired data: The pilot lacks per-example correctness, so its claims remain unpaired and Eq. 2 uses a paired-disagreement upper bound rather than measured ρ̂d.
  • Method coverage: Only NF4 via BitsAndBytes is tested, while GPTQ, AWQ, SmoothQuant, and SpQR may have different reliability landscapes.
  • Model coverage: The 7B tier contains only two models, making within-7B claims observational rather than broadly supported.
  • Prompt coverage: Prompt variance is measured only on MMLU at n=50, limiting firm interaction claims across benchmarks and templates.
  • Statistical scope: Eq. 2 is a conservative sufficient normal-approximation bound, not an impossibility result, and its assumptions can fail under MMLU subject clustering.Small-m regimes may require exact conditional or mid-p McNemar methods, while the paper did not run a subject-stratified bootstrap.

10. Conclusion

The paper reframes quantization reliability as a minimum-detectable-effect problem and uses a paired MDE bound to budget benchmark size against claimable effects. Its audit finds substantial binomial sampling noise and shows that prompt-template variance can match or exceed quantization deltas.

  • The paper derives a paired MDE bound that makes the smallest claimable quantization effect depend on benchmark size and planning assumptions.
  • 25 of 32 observed cross-split standard deviations fall within ±1.5 pp of the binomial reference.
  • At n=100 per split, the audit cannot resolve sub-percentage-point effects under the stated planning setup.
  • MMLU prompt-template variance ranges from 2–10 pp and meets or exceeds the largest observed quantization delta of 3.2 pp.
  • The paper recommends making minimum detectable effects explicit to encourage more rigorous statistical practice for quantized-model comparisons.

A. Reproducibility

The audit uses fixed model, dataset, split, prompt, scoring, and reporting configurations to compare quantized and FP16 evaluations. It reports per-cell accuracy variability, residuals against binomial references, and direct MDE comparisons.

  • Experimental setup: Four models are evaluated on MMLU, ARC-Easy, WinoGrande, and HellaSwag using five non-overlapping splits of 100 items.
  • Experimental setup: The default benchmark sweep uses the T0 prompt template, while MMLU prompt sensitivity is separately measured across three templates.
  • Scoring: Per-token candidate likelihood selects multiple-choice answers, while WinoGrande uses a joint likelihood ratio for its two continuations.
  • Variance diagnostics: 25 of 32 cells have cross-split SD residuals within ±1.5 pp of the binomial reference, and 29 are within ±2.0 pp.
  • MDE comparison: Only OPT-WinoGrande exceeds the ρd=0.05 MDE at 3.2 pp, whereas no cell crosses the approximately 4.0 pp MDE at ρd=0.10.

D. Wilson Confidence Intervals

The paper uses Wilson intervals to contextualize FP16 accuracy estimation, while emphasizing that paired confidence intervals would be the relevant uncertainty measure for FP16–NF4 deltas. It also reports paired-MDE planning quantities and QRI table definitions.

  • Wilson intervals: At n=500, Wilson 95% half-widths for FP16 accuracy range from ±3.7 to ±4.4 pp across the reported accuracy range.
  • Wilson intervals: At n=100 per split, Wilson half-widths are roughly ±8–10 pp, making split-level deltas less resolvable than union-level deltas.
  • Interval limitation: The reported Wilson intervals are single-proportion intervals for FP16 accuracy, not paired intervals for the NF4−FP16 delta.
  • Interval limitation: The paired CI would generally be tighter because FP16 and NF4 correctness are positively correlated, but per-example records were not retained.
  • MDE planning: A paired MDE at α=0.05 and 80% power scales as 2.80√(ρd/m), with required sample sizes reported for selected effect and disagreement-rate pairs.
  • QRI diagnostics: The QRI table reports split variability for all cells and prompt variability only for the four MMLU cells where prompt measurements were collected.
Loading 2605.28873v1…