PaperlayerView exact revision on arXiv

Source-linked AI summary

Certifying Compressed Language Models: An Audit and a Statistical Toolkit

Amogh Singh

arXiv:2608.15046v1cs.LG

TL;DR

Aggregate accuracy differences can obscure opposing item-level changes, while equivalence claims often omit declared margins, tests, and sufficient sample sizes. This paper develops paired margin-based equivalence certification and finds roughly fivefold churn-to-net-delta across 1,707 cells; none of 16 eligible audited claims declared a numerical tolerance or released task-matched per-item outputs.

  • Problem

    Compressed-model equivalence is often supported by aggregate accuracy differences, while declared margins, tests, and sufficient sample sizes are usually absent; net deltas also cancel opposing item-level changes.

    Source: §1 · p. 2

  • Method

    The paper develops paired TOST certification at a declared margin and tables of required evaluation sizes indexed by benchmark family and compression-disagreement scenario.

    Source: §3.1 · p. 8, §3.3 · p. 10

  • Results

    Across 1,707 paired cells, per-item churn was roughly five times the net delta; among 16 eligible audited claims, none declared a numerical tolerance or released task-matched per-item outputs.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The proposed reporting standard applies beyond compressed-versus-original comparisons to any pair of models similar enough to merit comparison, using per-item evidence and a prespecified margin.

    Source: §10 · p. 35

    Audit proportions describe only 16 eligible sources from three fixed frames and are not estimates of reporting-practice prevalence in compression literature.

    Source: §9 · p. 33

Abstract

from arXiv · show

A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original. That quantity is least informative when two models are most alike: a net delta is what survives cancellation between opposing per-item changes, and cancellation is most complete in the regime equivalence claims occupy. Across an atlas of 1,707 paired model-by-task cells mined from public per-item evaluation dumps (1.3B-405B), churn runs roughly five times the net accuracy delta, and cells scoring identically to their baseline still disagree on individual items. In a preregistered audit of 17 equivalence claims from three registered frames (method papers, model cards, vendor documentation), 16 are eligible. None states a prospective numerical equivalence margin, and none releases task-matched per-item outputs, though 3 release outputs for other tasks only; 5 report too little to assess numerically, so a reader cannot check them at any sample size. We audit evidential sufficiency, not truth: no claim is called false. We supply the missing instrument: paired equivalence testing at a declared margin, with certification tables giving the items an evaluation needs, computed from disagreement observed under compression, not from independent-binomial variance. A controlled experiment pairs GPTQ and AWQ on byte-identical calibration samples across five seeds. Under the frozen eight-cell decision rule H3 is supported: changing the calibration draw was sufficient to reverse the observed method ordering in 5 of 8 confirmatory cells. The reporting standard we propose is five lines: declare a margin, run the paired test, report churn beside net delta, cite the sample size you met, release per-item outputs. It applies to any comparison between two models alike enough to be worth comparing. All per-item outputs, protocols and code are released.

1 Introduction

Aggregate accuracy can conceal substantial per-item disagreement, making it weak evidence of equivalence precisely when models appear most alike. The paper measures that cancellation in public evaluations and controlled comparisons, audits whether published claims can be checked, and proposes paired equivalence testing and a five-line reporting standard.

A The aggregate view

  • In one 1,000-item GSM8K comparison, a 0.58-point gap concealed 9.1% correct-to-wrong and 8.5% wrong-to-correct changes; certifying a ±2-point margin required 2,730 items.That planning requirement means the evaluation cannot support the equivalence claim, not that the methods differ.Source: §1 · p. 2, B The same items, paired · p. 3

B The same items, paired

  • Of 16 eligible audited claims, none declared a numerical margin or released task-matched per-item outputs; 3 released outputs for other tasks, and 5 could not be assessed numerically.The audit evaluates evidential sufficiency, not truth; no claim was called false.Source: §1 · p. 2, B The same items, paired · p. 4
  • Across 1,707 paired cells, median per-item churn was about five times median net accuracy delta; 128 of 145 identical-accuracy cells still changed which items they got right.The atlas was mined from public per-item evaluation dumps and is observational, not a causal claim about quantization generally.Source: §1 · p. 2, B The same items, paired · p. 3
  • In 5 of 8 controlled cells, changing the calibration sample reversed the observed GPTQ–AWQ ranking under the frozen decision rule.The methods were paired on byte-identical calibration samples across five seeds; observed churn had a median of 12.7× the net delta.Source: Figure 1 · p. 3, B The same items, paired · p. 4
  • The paper provides paired equivalence tests and certification tables that calculate required item counts from disagreement observed under compression, rather than independent-binomial variance.The proposed shift is from detecting a difference to certifying equivalence at a declared margin using TOST and required-n tables.Source: B The same items, paired · p. 4
  • On one fixed FP16 model, changing the answer-extraction filter moved reported GSM8K accuracy from 0.232 to 0.566 without changing model output.This exploratory result concerns one model and one task, and reflects a scoring decision rather than a modeling change.Source: B The same items, paired · p. 4
  • The proposed five-line standard calls for a declared margin, a paired test, churn alongside net delta, the sample size met, and released per-item outputs.The lines are intended to make equivalence claims testable and their evidence checkable by third parties.Source: B The same items, paired · p. 5

2 Related work and positioning

The paper builds on prior work documenting per-item flips and calibration sensitivity, while positioning its contribution as a public-record decomposition, churn-based equivalence certification, and audit of reported claims. It also illustrates the reporting standard using an existing evaluation, not a new experiment.

2 Related work and positioning

  • Dutta et al. [2024] proposed flips and KL divergence for aggregate-matched models; this paper adds a public-record decomposition, churn-based certification variance model, and audit.Rababah et al. [2026] independently study correctness agreement over a disjoint quantization family; the paper treats this as corroboration, not competition, and makes no priority claim.Source: §2 · p. 5

B The same evaluation, reported to the standard

  • Across 16 eligible audited sources, 0 declare a prospective numerical margin and 0 release task-matched per-item outputs.Column B reports quantities already produced by the run summarized in column A; it adds no experiment, and column A has no further evidence behind it.Source: B The same evaluation, reported to the… · p. 6
  • For one 1,000-item Qwen2.5-7B GSM8K cell, 4-bit GPTQ versus 4-bit AWQ has a 0.58 pp net delta, while the ±2 pp planning requirement is 2,730 items.The 2,730 figure assumes a true difference of zero and does not show that the methods differ; the reported bootstrap interval is not the TOST interval, so the cell has no TOST verdict.Source: Figure 2 · p. 6, B The same evaluation, reported to the… · p. 6
  • Prior work finds calibration-set effects, but this paper asks whether changing only the calibration seed reorders two methods, a question the sweep found unstudied.Across the paper’s eight cells, the seed-induced range is at least as large as the gap between methods in 7 of 8; Williams and Aletras [2024] find substantial set effects, while Paglieri et al. [2024] find diminishing effects on modern models.Source: B The same evaluation, reported to the… · p. 7
  • Kübler et al. [2026] use one-sided McNemar testing to detect differences, while this paper uses TOST at a declared margin to certify equivalence.A non-significant detection result is not equivalence and can reflect inadequate resolution; the paper also computes required-n tables from empirical churn rather than independent-binomial variance, which implies a 2.3–14.7× correction.Source: B The same evaluation, reported to the… · p. 7
  • Helcig et al. [2026] define quantized-LLM losslessness, complementing this paper’s question, but do not compute an equivalence test, power, or required sample size.The paper follows constructive-audit precedents [Dodge et al., 2019, Marie et al., 2021] and assesses reported evidence rather than whether individual authors or methods are right.Source: B The same evaluation, reported to the… · p. 7

3 Paired certification: what an equivalence claim has to show

Paired equivalence certification asks whether the difference between a compressed model and its baseline falls within a margin declared in advance—not merely whether a difference is detectable. Its required sample size depends on measured item-level disagreement, and meeting that count makes a test informative but does not certify equivalence by itself.

3.1 Detection is not certification

  • TOST tests whether the paired difference lies within a prespecified ±m margin; unlike a failed difference test, it can provide evidence of equivalence.TOST uses two one-sided tests at level α each; the design table assumes a true difference of zero, so its required item counts are lower bounds when the true difference is nonzero.Source: §3.1 · p. 8, §3.2 · pp. 8–9, Table 1 · p. 9

3.2 Method

  • The paired sample-size calculation is driven by item-level discordance, while the independent-sample calculation uses median baseline accuracy.The calculation assumes a true paired accuracy difference of zero; resolving a nonzero difference requires more items, without bound as its magnitude approaches the margin.Source: §3.2 · pp. 8–9, Table 1 · p. 9

3.3 Reading the table

  • 2,164 items are required for MMLU at a 2 pp margin and typical discordance, versus 7,727 if pairing is ignored; the paired design cuts that requirement 3.6×.For MMLU, tightening the margin to 1 pp raises the requirement to 8,656 items, while relaxing it to 3 pp lowers it to 962.Source: §3.3 · p. 10

3.4 The requirement is driven by churn, not by difficulty

  • MMLU requires 2,164 items at a 2 pp margin, versus 749 for harder GPQA; their typical discordance rates are 0.140 and 0.048, respectively.The requirement follows item-level churn, not benchmark difficulty or baseline accuracy.Source: §3.4 · p. 10, §3.6 · p. 12, Figure 3 · p. 12

3.5 Which benchmarks this fits, and how to size an evaluation

  • The framework applies to benchmarks with binary per-item outcomes; graded metrics need an unvalidated continuous adaptation, and binary benchmarks without atlas rows require measuring discordance on the practitioner’s model pair.Eleven benchmark families have measured discordance rows; HumanEval and MBPP are covered by the arithmetic but lack supplied discordance rates.Source: §3.5 · pp. 10–11

3.6 Scope and caveats

  • The reported counts are planning sizes, not certification: nonzero true differences require more items, and the framework does not support extrapolation to absent families or compression methods.Families with fewer than four analysable atlas cells are omitted; meeting a count only makes a paired equivalence test informative, and that test must still be run and pass.Source: §3.6 · pp. 11, 13, Figure 3 · p. 12

3.7 Preregistration compliance, and the choice that moved the headline

  • A preregistration ruling was corrected: the uniform 2 pp margin left 1 of 11 assessable claims below the planning threshold, after earlier readings had produced counts of 5 of 12 and 4.The authors state that no audited source specified a margin, so the claim-specific condition did not apply; Amendment 2 withdrew the earlier implementation.Source: §3.7 · p. 13

3.8 Atlas validation and population correction

  • 262 fields across ten stratified cells reconciled in an independent spot-check, but two defects had non-randomly dropped cells; the repair enlarged the analysable population by 44%.The repair followed inspection of results, left the audit’s headline verdicts unmoved, and both population revisions were published.Source: §3.8 · p. 14

4 The atlas: cancellation at scale, and what certification costs

Across the public atlas, per-item behavior changed much more than net accuracy deltas suggest, and many evaluations could not establish equivalence at the registered margin. These findings describe the two sources in the atlas, not quantization methods or models in general.

4.1 Construction, coverage, and the analysis population

  • Of 2,055 enumerated pair-task cells, 1,807 passed pipeline accounting, but the headline analysis uses 1,707 cells after excluding 99 disclosed feasibility probes.An empty join means the pair was evaluated on different item sets, so its accuracy difference is not paired and cannot support per-item analysis; aggregate leaderboard numbers do not reveal this.Source: §4.1 · pp. 14–15

4.2 Net delta versus per-item churn

  • Median accuracy-state churn was 5.40 times median absolute accuracy change across 1,707 cells, with similar ratios in S1 (5.22) and S2 (5.19).From S1 to S2, median churn fell by a factor of 2.86 and net delta by 2.85, leaving the understatement ratio nearly unchanged.Source: §4.2 · pp. 15–16, Table 2 · p. 16
  • At the registered 2 pp margin, 69.2% of S1 and 77.7% of S2 cells were neither certifiable as equivalent nor detectably different at observed sample sizes.S1 had 4.9% certifiable and 26.5% detectably different cells; S2 had 17.2% certifiable and 6.1% detectably different cells. Smaller margins require more items, not fewer.Source: Table 2 · p. 16, §4.2 · pp. 16–17

4.3 Identical scores, different answers

  • Of 145 cells with exactly identical baseline accuracy, median accuracy-state churn was 0.0720, and 128 had nonzero churn.In the most extreme example, 17.17% of items broke and 17.17% healed, yielding zero net delta and exact McNemar p = 1.0 despite a third of answers changing; this S1 cell illustrates the mechanism, not a typical magnitude.Source: §4.3 · p. 17

4.4 Population caveats

  • The atlas describes the public reporting record, not quantization on average: S1 is coverage-selected community work, while S2 is one vendor’s releases evaluated by that vendor.The section does not establish causal attribution or support general claims about absent models or methods; per-cell item counts inherit the original evaluations’ often-small sample sizes.Source: §4.4 · pp. 17–18

5 What published equivalence claims actually report

The audit assesses whether reported evaluations support published equivalence claims, not whether the claims or models are correct. Across 16 eligible claims, none states a prospective numerical margin or releases task-matched per-item outputs, and the lone planning-threshold flag is sensitivity-dependent.

5.1 What was audited, and what was not

  • The audit covers 16 eligible claims from three registered frames and judges evidential sufficiency, not truth; it does not call claims false or estimate field-wide prevalence.One of 17 candidates was excluded under the registered rule because its qualifying quotation appeared nowhere in its source.Source: §5.1 · pp. 18–19

5.2 What the sources declare

  • No audited source declares a prospective numerical equivalence margin: 10 of 16 make qualitative claims, while the other 6 report only measured outcomes.Because the sources set no margin, the audit evaluates them using its registered margin rather than treating any reported outcome as a decision rule.Source: §5.2 · pp. 19–20

5.3 Where the claim is written

  • The 17 captured claims are a floor, not a census: claims across six model cards varied from prose to score juxtaposition to a table cell, and the inclusion rule misses some.The audit's reported quantities apply to captured sources, so the true rate of unbounded equivalence claims is understated rather than overstated.Source: §5.3 · p. 20

5.4 Availability of per-item outputs

  • 0 of 16 eligible sources release task-matched per-item outputs; three provide outputs only for other tasks, preventing third parties from rerunning paired tests or computing churn.The three partial cases are R08, R15, and R16; without task-matched outputs, the comparison cannot be checked downstream at any sample size.Source: §5.4 · p. 21

5.5 Verdict rules

  • The verdict protocol applies a registered 2 pp margin and estimates paired-test requirements using imputed discordance, classifying claims by whether reported sample size meets the requirement.Discordance rates are imputed from the nearest atlas cells and sensitivity is checked at their first and third quartiles.Source: §5.5 · pp. 21–22

5.6 Results

  • 1 of 11 assessable claims fell below the 2 pp planning threshold under median discordance imputation, but the sole flag, R01, did not survive sensitivity analysis.R01 reports n = 1,838 against a requirement of 2,010 at an imputed discordance rate of 0.130; its classification reverses at 0.1189. No claim falls below the threshold throughout the sensitivity interval.Source: Table 4 · p. 22, §5.6 · pp. 22–23

5.7 Claims that cannot be assessed

  • 5 of 16 claims are not numerically assessable: four lack registered inputs and one, R04, reports a quantity outside the registered paired binary-outcome calculation.R14 is excluded for lacking a baseline, despite its imputed sample size of 728 being close to the 742-item requirement; two insufficiently reported claims rely on chart images without extractable numbers.Source: §5.7 · pp. 23–24

5.8 R04: outside the registered calculation

  • R04's qualifying claim concerns COCO CIDEr, a graded metric without per-item correct/incorrect states, so the registered flip-model calculation cannot assess it.Paired resampling could assess a graded metric, but that extension was not part of the registered model; the audit's findings concern evidence, not model degradation or claim truth.Source: §5.8 · pp. 24–25

6 Cancellation is worse where practitioners choose

In the registered eight-cell GPTQ–AWQ comparison, calibration-seed changes make method rankings unstable, while cancellation makes per-item movement especially easy to miss when comparing similarly accurate methods. The evidence is strongest on MMLU and less resolved on most GSM8K cells at the sample size used.

6.1 Design

  • Identical ordered calibration samples per seed make a ranking difference attributable to method-by-calibration interaction, not different data exposure.Each seed uses 128 C4 samples of 2,048 tokens, and selected document indices and token hashes are persisted. The confirmatory design covers four models and two benchmarks across five seeds.Source: §6.1 · p. 25

6.2 The eight-cell verdict

  • 5 of 8 cells show winner flips across seeds, supporting H3 under the frozen rule; the range/gap criterion holds in 7 of 8.The supported limb is a disjunction, and each criterion independently satisfies one disjunct; the registered rule was applied once to all eight cells. No exact accuracy ties occurred, and the verdict is limited to 4-bit GPTQ versus AWQ in the registered cells.Source: §6.2 · p. 26

6.3 Supporting analyses

  • On MMLU, seed-level SD is 1.6 to 5.9× item-level SE in every cell, while GSM8K’s two variance components are of the same order.Winner flips occur in 5 of 8 cells, but the bootstrap rank-flip rate is below 0.05 in 6 of 8; the measures differ because the bootstrap tests whether the five-seed mean ranking survives joint resampling. Five seeds provide limited bootstrap resolution and are not established as sufficient; two GSM8K cells are unstable on both measures.Source: §6.3 · pp. 27–28

6.4 Cancellation is more complete between two methods than against FP16

  • 12.7 versus 3.85 is the median per-cell churn-to-net-delta ratio in the eight controlled cells and across the atlas, respectively.The ratio-of-medians comparison is 12.1 in the controlled cells versus 5.40 in the atlas. In Qwen2.5-7B/GSM8K, methods differ by 0.58 pp in aggregate accuracy while 17.7% of items change correctness state and 28.7% of answers change outright; these are two measured regimes, not evidence of a monotone curve.Source: Table 6 · p. 28, §6.4 · pp. 28–29

6.5 Which arm of the experiment carries the result

  • All four MMLU cells meet the range/gap criterion, with seed-induced ranges of 5.5 to 17.5 paired SE; GSM8K is under-resolved at n = 1,000 in three of four cells.The exception is Qwen2.5-1.5B, whose 9.68 pp GPTQ–AWQ gap is 5.65 paired SE and resolved under the same benchmark, sample size, and test. The registered verdict remains based on the per-seed criteria, not a significance test.Source: §6.5 · pp. 29–30

6.6 Resolution of the confirmatory cells against the certification tables

  • At n = 1,000, confirmatory GSM8K cells fall below the approximately 1,184-item median-discordance requirement for a 2 pp margin.Observed discordance is pd = 0.1766, 0.1808, 0.2086 and 0.2940, at or above the certification table’s p75 bracket, where the requirement is roughly three times the sample used. The percentile labels are from a quantized-against-FP16 contrast, whereas these cells compare GPTQ with AWQ.Source: §6.6 · p. 30

7 Exploratory scoring-pipeline sensitivity

This exploratory study finds that evaluation-harness configuration can move per-item correctness on a fixed model by amounts comparable to quantization, and sometimes more. Its findings are limited to one model and bridge item subsets and do not decide anything about compression methods.

7 Exploratory scoring-pipeline sensitivity

  • A live scoring-pipeline defect moved reported GSM8K accuracy from 0.232 to 0.566 without changing any model-output token.The campaign found two live configuration defects on pinned lm_eval 0.4.12 within eight days.Source: §7 · p. 30
  • R compares configuration churn on the fixed FP16 model with mean churn across ten Qwen2.5-1.5B quantized variants: 0.199 on MMLU and 0.287 on GSM8K.The protocol reports both inputs alongside every ratio.Source: §7 · p. 31
  • Across five conditions, R ranged from 0.836 to 1.585; the scoring-filter choice reached R = 1.585, exceeding the quantization churn denominator.On the fixed FP16 model, configuration changes moved per-item correctness by as much as or more than swapping in a quantized variant.Source: §7 · p. 31
  • The study covers Qwen2.5-1.5B-Instruct on bridge subsets of 400 MMLU and 200 GSM8K items; no Llama R is reported because its seed-0 canary has not run.The study is exploratory, adjusts no confirmatory gate or analysis, and states no H3 outcome.Source: §7 · p. 31

8 Artifacts

The section documents six released artifacts and explains how the frozen release preserves the audit and makes its materials reproducible despite restrictions on republishing captured source texts.

8 Artifacts

  • Six artifacts are released, including 88 controlled-experiment per-item JSONL files, a flip atlas, flipeval, the frozen audit table and verdict CSV, certification tables, and a reproduction package.The per-item outputs address a gap identified by the paper’s audit: none of the 16 eligible sources released task-matched per-item outputs.Source: §8 · p. 31
  • Because redistribution review barred release of captured full texts, the archive instead provides source URLs and pinned versions, SHA-256 hashes, a manifest, and a retrieval script.The review found four sources without a grant permitting third-party republication; seven method papers used arXiv’s default licence, which authorises arXiv rather than the authors.Source: §8 · pp. 31–32
  • The canonical release is Zenodo 10.5281/zenodo.21939143, the frozen v1.2.0 state; v1.0.0 remains published as a historical record of corrected audit verdicts.The v1.2.0 package is tagged on GitHub and mirrored on Hugging Face; v1.0.0 predates the rev-3 verdicts and contains a superseded rev-2 file.Source: §8 · p. 32

9 Limitations

The atlas and audit characterize a bounded record of circulating evidence, not quantization generally, literature-wide prevalence, or whether audited models are truly equivalent. The controlled experiment and certification method also have narrow scope: the experiment covers eight registered cells, while graded-metric certification is described but not validated here.

9 Limitations

  • Atlas statistics describe circulated evidence, not quantization generally; audited sources reported no per-item disagreement rates, so the audit imputed them from the nearest nonempty atlas tier.The atlas is rev-2 after two independent spot checks; coarser matches rely on wider, less similar sets of cells, which are reported per claim.Source: §9 · p. 32
  • The audit covers 16 eligible sources in three fixed frames and assesses evidential sufficiency, not equivalence truth or prevalence across compression literature.Sources outside the frames and claims discovered after the table freeze are not included in these proportions.Source: §9 · p. 33
  • The causal experiment covers only eight registered cells (1.5B to 8B, two benchmarks, 4 bits, two methods, five seeds), so it licenses no broader model or benchmark claims.Its bootstrap resamples seed labels from only five seeds; low rank-flip rates bound stability of the observed five-seed mean, not the seed-level distribution itself.Source: §9 · p. 33
  • At n = 1,000, GSM8K fell short of estimated needs (1,184 items at median discordance; 3,068 at p75), leaving MMLU as the evidence-bearing cells.The GSM8K shortfall was predicted before measurement and does not affect the frozen-rule verdict, which is a count rather than a significance test.Source: §9 · p. 33
  • The registered flip model covers English correctness on multiple-choice or short-answer benchmarks, excluding 33 float-scored atlas cells and quality claims about open-ended generation.The exclusion is a limit of the registered flip model, not paired analysis in general.Source: §9 · pp. 33–34

9.1 Continuous and graded metrics, and what certifying them would take

  • TOST extends to bounded graded per-item scores, but this paper validates no such extension; metric-specific variance and margins are required, and binary required-n values do not transfer.For graded metrics, churn becomes dispersion of score changes and the margin must use the metric’s own units.Source: §9.1 · p. 34

10 Conclusion: a reporting standard for compression claims

Aggregate accuracy can hide substantial item-level changes between similar models. The conclusion calls for checkable comparisons grounded in per-item evidence and a margin declared before evaluation.

10 Conclusion: a reporting standard for compression claims

  • 17.7% of items differed between two 4-bit Qwen2.5-7B quantizations just 0.58 points apart on GSM8K; ±2-point certification needed 2,730 items, but evaluation ran 1,000.Which quantization came out ahead changed with the calibration draw; this was one evaluation and the most extreme of the eight registered cells.Source: §10 · pp. 34–35
  • Among 16 eligible audited sources, none declared a prospective numerical equivalence margin or released task-matched per-item outputs; 3 released outputs for other tasks only, and 5 could not be assessed numerically.These are counts for the audited sample, not estimates for the literature.Source: §10 · p. 35
  • Equivalence certification at a declared margin uses sample-size tables computed from observed churn rather than assumed variance; an extension to anytime-valid sequential certification is registered but unrun.No results are reported for the sequential component.Source: §10 · p. 35
  • 12.7× was median per-cell churn relative to net delta for GPTQ-versus-AWQ compressed models, versus roughly 4× in the atlas; calibration-seed changes reversed the winner in 5 of 8 registered cells.The comparison places two compressed models at the same bit width side by side.Source: §10 · p. 35
  • The standard extends beyond compression to comparisons between similar models, including method, checkpoint, and version comparisons, using per-item evidence and a margin declared before evaluation.The conclusion argues that aggregate accuracy is least informative in the regime where such comparisons are made.Source: §10 · p. 35

A Reconciling the calibration-sensitivity antecedents … B.4 The H3 reporting rule, stated before the results exist

Calibration robustness in absolute accuracy does not establish stable rankings between methods; this paper argues the distinction explains apparently conflicting antecedents. Its preregistered design fixes the H3 decision rule and analysis procedures before results are inspected.

A Reconciling the calibration-sensitivity antecedents

  • Williams and Aletras [2024] find calibration-set sensitivity while Paglieri et al. [2024] find modern models robust; this study instead varies the seed within a fixed corpus and tests method rankings.The antecedents vary calibration data but study one method at a time; neither measures whether the ordering between methods survives.Source: A Reconciling the… · pp. 38–40
  • In 7 of 8 confirmatory cells, seed-induced variation was at least as large as the mean GPTQ–AWQ gap, indicating ranking instability despite individual-method stability.In a post-hoc resolution analysis, MMLU ranges were 5.5–17.5 paired standard errors at n = 14,042; GSM8K ranges were roughly two standard errors at n = 1,000 and did not resolve the effect.Source: A Reconciling the… · pp. 39–40

A.1 Adjacent concurrent measurements / A.2 Losslessness definitions and fidelity metrics

  • Adjacent work measures per-item model changes or checkpoint flip decompositions, while Helcig et al. [2026] define losslessness; this paper instead audits claims and supplies sample sizes for certification.Expected Acceptance Rate can serve as a fidelity target, but the paper argues that the accuracy component still needs an evaluation large enough to certify it.Source: §A.1 · p. 40, §A.2 · p. 41

A.3 The constructive-audit genre, and preregistration precedent

  • The paper follows constructive-audit precedents in focusing on reporting practice rather than whether individual authors are right, and notes preregistration precedents in evaluation research.It names Dodge et al. [2019], Marie et al. [2021], Gringras and Salahshoor [2026], and Thomas et al. [2026].Source: §A.3 · p. 41

A.4 The compression families the audited claims span

  • The audit covers equivalence language and evidence across weight-only, weight-and-activation, and one-shot-pruning methods, not whether the methods themselves work.SpinQuant [Liu et al., 2025] and QuIP# [Tseng et al., 2024] are described as reporting accuracy deltas without equivalence language.Source: §A.4 · pp. 41–42

B Preregistration detail / B.1 The freeze timeline

  • The preregistration appendix collects the freeze timeline and related safeguards; Table 7 records that artifacts were committed before governed analyses and amendments were appended.Hashes for the frozen claim table and atlas summary are also recorded so readers can check the inputs to reported numbers.Source: B Preregistration detail · p. 42, Table 7 · p. 42, §B.2 · p. 42

B.2 What the freezes fix in advance / B.3 Result-inspection discipline

  • Audit extraction, verdict formulas, robustness sweeps, and atlas pairs were fixed before their respective analyses, while accuracy inspection and registered analysis followed a controlled sequence.The paired-bootstrap tie convention was registered in advance, and no registered analysis was tuned after results were seen.Source: §B.2 · p. 43, §B.3 · p. 43

B.4 The H3 reporting rule, stated before the results exist

  • The frozen H3 rule tests eight 4-bit GPTQ–AWQ seed comparisons and supports H3 at specified flip or range/gap thresholds; all eight cells completed for the single full-rule application.The mini-grid registration specified that four completed cells alone would leave H3 undecided; the remaining cells were built after the mechanical escalation screen fired.Source: §B.4 · pp. 43–44

B.5 Disclosed pre-registration data contact … D.4 Discordance imputation

The appendices disclose how pre-registration contact, interpretive rulings, and an atlas population correction changed or qualified the audit, while documenting the methods and data needed to inspect the results. They also show that the corrected atlas changed descriptive estimates but not the headline verdicts then reported.

B.5 Disclosed pre-registration data contact

  • 145 of 1,707 cells (8.49%) scored identically to baseline at a median churn of 7.20%, the registered replacement for a retired anecdote.The anecdote involved roughly 6.3% item flips with unchanged aggregate score, but was excluded from quantitative use as pre-registration data and outside the frozen manifest; two atlas probes were likewise disclosed and all 99 cells from their pairs excluded from headline aggregates.Source: §B.5 · p. 44

B.6 The interpretive choices that moved the headline

  • 1 of 11 assessable claims fell below the approximate planning threshold after Amendment 2 restored the registered 2 pp margin.The superseded analysis substituted reported deltas for prospective margins; paired variance and median discordance imputation were retained as choices resolved in the source’s favour.Source: §B.6 · pp. 45–46

B.7 Atlas validation and population correction

  • The independent spot-check reconciled all 262 fields, yet found that two implementation defects had produced a biased atlas population.The code omitted required reverse-chronological run fallback for 11 pairs and missed binary metrics in 583 cells; the repair followed the frozen protocol, and both revisions were published.Source: §B.7 · pp. 47–48

B.8 The rev-1 to rev-2 delta

  • S1 median churn rose from 0.1327 to 0.1375, while the share of cells certifiable at 2 pp fell from 5.6% to 4.9%; the headline verdicts did not move.The probe-excluded analysable population grew from 1,155 to 1,707 cells, a 44% enlargement; the unchanged verdict counts cited in this section were historical, not current results.Source: §B.8 · p. 49

C Full audit table

  • The released audit tables map 17 claims and report per-claim characterisation, sensitivity, and power quantities at the registered 2 pp margin.The CSV records claim eligibility and evidence, imputed discordance and matching tier, assessability, paired and independent-binomial MDDs, required sample sizes, and threshold robustness; non-assessable-row numbers are not verdicts.Source: C Full audit table · pp. 49, 50, 52, Table 8 · p. 50, Table 9 · p. 51

D.1 Where the equivalence claim is written

  • Six cards from one publisher present identical evidence but place equivalence claims differently, and the prose-keyed inclusion rule cannot capture every claim.Three make prose assertions, two juxtapose scores without characterising the gap and were retained, and one relies only on a recovery table cell and was excluded as R10.Source: §D.1 · pp. 53–54

D.3 The three registered verdict quantities

  • The registered verdicts comprise paired-model detection resolution, TOST equivalence support, and a binary check for task-matched per-item output reproducibility.TOST uses a one-sided z of 1.6449 at α = .05 per bound, requiring a 90% two-sided confidence interval inside ±m; independent-binomial bounds and 1 pp and 3 pp margin sweeps are also computed.Source: §D.3 · pp. 54–55

D.4 Discordance imputation

  • The imputed discordance rate uses the median from the most-specific non-empty matching atlas tier, with pruning claims R06 and R07 assigned at tier 4.Their 183 matching cells are 4-bit quantization cells with missing parser profile labels, so the match rests on missing bit width rather than method similarity; both claims remain above threshold even at the third quartile.Source: §D.4 · pp. 55–56

D.5 The two robustness readings … E Certification tables at 1 pp and 3 pp

Under the registered 2 pp margin, only R01’s evaluation is coarser than the margin; the appendix also documents how eligibility, source reporting, and two distinct robustness sensitivities qualify the audit. Certification requirements vary quadratically with the margin.

D.5 The two robustness readings; D.12 Interpretive rulings and superseded verdicts

  • Amendment 2 sets the applicable margin at 2 pp for every claim, while margin and imputation sensitivity remain distinct robustness questions.R01 flips across the 1 pp to 3 pp margin sweep and across the imputed discordance-rate range, reversing at d = 0.118915; R04 and R14 also flip across margins but have no classification to qualify. Claim-derived margins and the shortfall range from the superseded reading are withdrawn.Source: §D.5 · p. 56, §D.12 · p. 59

D.6 Eligibility, and the full-text source review; D.7 The complete-text margin sweep

  • A full-text review excluded R10 under the registered rule, moving the eligible denominator from 17 to 16; no archived source declares a prospective numerical equivalence margin.The review searched all source content, including tables, captions, footnotes, appendices, and reference lists. The complete-text sweep found no tolerance threshold: margin occurrences were layout artefacts or unrelated uses, and one vendor document explicitly declined to fix a tolerance.Source: §D.6 · p. 57, §D.7 · p. 57

D.8 Per-item outputs: what the three partial sources release

  • Three sources release per-item outputs for other benchmarks, but none releases task-matched outputs for the OpenLLM v1 tasks underlying its audited claim.Without per-item outputs, outsiders cannot run paired tests, compute churn, or check arithmetic at any sample size; the paper notes that a few thousand benchmark items require only a few megabytes of output.Source: §D.8 · p. 58

D.9 The non-assessable claims, one by one

  • Four claims are non-assessable for insufficient reporting, while R04 is outside the registered calculation because its reported quantity cannot be scored by the paired-outcome model.R02 and R11 lack sample size, baseline, and numeric delta; R13 gives n = 250 without an on-page baseline; R14 reports 0.7 pp but no baseline. R04 reports enough, but about a quantity the registered model cannot score.Source: §D.9 · p. 58

D.10 R01: the planning values behind the single flag

  • R01 reports n = 1,838 against a 2,010-item requirement at imputed discordance 0.130, and its classification reverses at 0.1189.Across the atlas-cell interquartile range, the requirement moves from 1,364 items to 4,328; R01 is adequately sized at the first quartile and undersized at the third.Source: §D.10 · pp. 58–59

D.11 R04: the overruled first-pass computation

  • R04’s first-pass GSM8K result, −0.30 pp at n = 1,319, was overruled because it tested a benchmark different from the source’s claim.The computation remains in the released CSV as a labelled transparency column, not as an audit result; the earlier largest-shortfall characterization is superseded.Source: §D.11 · p. 59

D.13 Both robustness directions in full; E Certification tables at 1 pp and 3 pp

  • Only R01’s evaluation is coarser than the registered 2 pp margin; ten others resolve differences between one-sixth and one-half of it.The independent-binomial columns are uniformly worse by roughly a factor of two. MMLU’s median requirement is 8,656 items at 1 pp, 2,164 at 2 pp, and 962 at 3 pp.Source: §D.13 · p. 60, E Certification tables at 1 pp and 3 pp · p. 60

E.1 Certification method detail … H.5 Why MMLU collapses to a single off-reference cell

The appendix details paired certification, atlas construction, and controlled evaluation audits, showing how pairing, sample scope, and scoring configuration affect what accuracy comparisons can establish. It distinguishes screening and supporting analyses from verdicts and reports exploratory evidence with explicit limitations.

E.1 Certification method detail

  • Paired certification uses disagreement-based variance rather than independent-binomial variance, with TOST at one-sided α = .05 corresponding to a 90% two-sided interval.Required-item tables use atlas-observed discordance quartiles; pairing gains vary by family, with low-churn MuSR, BBH, and GPQA benefiting more than high-churn MATH and MMLU.Source: §E.1 · pp. 60, 62, 63, Table 13 · p. 61, Table 14 · p. 62

F.3 Exclusion breakdown

  • The atlas pairs items by identical prompt hashes and analyzes 1,807 of 2,055 cells, while mmlu_pro and ifeval remain thin at 5 and 8 cells.The sources are leaderboard quantizations and Neural Magic/Red Hat dumps; 248 cells were excluded or skipped, and probe pairs with discordance up to 0.9 were excluded. Family quartiles are conservative because aggregation mixes subject- and model-level variation.Source: §F.1 · p. 63, §E.2 · p. 63, §F.3 · p. 64

G.1 The mechanical escalation screen

  • Winner flips occurred in 2 of 4 cells, and the range/gap criterion held in 3 of 4, so the screen fired and authorized construction and evaluation of deferred 7B/8B seed cells.GPTQ and AWQ received byte-identical calibration samples at each seed, and the four cells remained sealed until the validator passed the complete 44-JSONL expected set (409/409 checks). This was not an eight-cell verdict, which was defined only when all eight cells existed.Source: §G.1 · pp. 64, 66, 67

G.3 The three registered supporting analyses

  • The registered supporting analyses describe variance, bootstrap, and flip statistics but cannot alter the verdict; the post-hoc resolution analysis was requested only after it was signed.The supporting tables distinguish item-level uncertainty from across-seed variation and separate post-hoc quantities from registered analyses.Source: §G.3 · p. 67, Table 18 · p. 68, Table 19 · p. 69

G.6 The earlier public-checkpoint pilot

  • The earlier public-checkpoint pilot detected a −4.25 pp MMLU difference at n = 400, while GSM8K at n = 200 correctly declined detection despite churn of 0.250 and 0.220.The pilot was exploratory, could not test H3, and had prompt, checkpoint, and interval limitations; registered 3-bit, benchmark, and calibration-distribution analyses remained deferred.Source: §G.6 · pp. 69–70, §G.5 · p. 69

H.1 Two observed defects

  • On unchanged Qwen2.5-1.5B generations, strict-match scoring moved reported accuracy from 0.232 to 0.566; the configuration study used FP16 bridge subsets, not confirmatory cells.The quantization-churn denominators were 0.199 for MMLU and 0.287 for GSM8K, computed only after the registered validator and authorized accuracy inspection.Source: §H.1 · p. 70, §H.2 · pp. 70–71

H.3 Condition B in full

  • 0.455 of GSM8K items changed correctness when the same generations were rescored, yielding R_B = 1.585 against quantization churn.All 91 changes were correct-to-incorrect, so the accuracy drop equaled churn; MMLU’s conditions C and D were byte-identical and were run once as C ≡D.Source: §H.3 · p. 71, §H.5 · p. 72

H.4 The same move, in another domain

  • Bronder [2026] held the player model fixed while changing four evaluation-instrument choices, paralleling this study’s design on a different object.The comparison supports examining instrument effects rather than assuming an evaluation is fixed.Source: §H.4 · pp. 71–72

H.6 Condition-by-condition results · I Artifact detail · I.1 The six released artifacts

Evaluation settings can move correctness at quantization-scale magnitudes, with opposite directions across tasks. The paper releases paired per-item outputs, analysis and audit tools, and source-retrieval records while withholding captured source text where redistribution was not permitted.

H.6 Condition-by-condition results

  • Three of four GSM8K conditions have R ≥1, while multiturn exemplar placement stays below the quantization scale.The conditions are turning off the chat template, accepting all stock defaults, and switching the extraction filter. Net accuracy deltas understate movement because they cancel directional churn; the stock-default condition rescores identical generations at zero GPU cost.Source: §H.6 · p. 73
  • +4.5 pp on MMLU (0.415 →0.460) from turning off the chat template, with 0.210 churn across 84 of 400 items and R = 1.055.The directional split is 33/51, meaning more items recovered than lost; the same setting worsens GSM8K, so its effect differs by task.Source: §H.6 · p. 73

I.1 The six released artifacts

  • 88 cell JSONL files provide per-item correctness for every controlled-experiment model × method × seed × task, while 0 of 16 eligible sources release outputs for their claimed tasks.Three sources (R08, R15, R16) release outputs for other tasks only; the files are loadable and also included in sealed archives with per-file SHA-256 checks.Source: §I.1 · pp. 73–74
  • The flip atlas supplies paired statistics for enumerated pair-task cells, and flipeval provides flip, churn, exact McNemar, TOST, bootstrap, and certification-table analyses.The package also includes item-bootstrap rank-flip rates, minimum detectable difference, required-n, and certification tables with twelve rows per margin at 1, 2, and 3 pp.Source: §I.1 · p. 74
  • The audit artifacts include the frozen 17-claim table, source content hashes, per-claim verdict CSV, robustness and transparency columns, source manifest, and retrieval script.Captured source text itself is not included.Source: §I.1 · p. 74
  • The reproduction package includes registrations and amendments, signed decision records, configs, container build materials, scheduler scripts, run manifests, freeze fingerprints, and an incident log.The incident log records 28 catches; the passage presents its inclusion as evidence that the pipeline was verified.Source: §I.1 · p. 74
  • The redistribution review found four sources without a grant permitting republication and seven method papers under an arXiv licence that did not authorise the authors to republish them.The captured corpus remains private; shipped source URLs, pinned versions, hashes, provenance statuses, and a retrieval script let readers rebuild and check it. R11 and R13 have stated limits on what their digests establish and are not described as hash-verified.Source: §I.1 · p. 74

I.2 Datasheet · I.3 Metadata and identifiers · I.4 Maintenance

The release makes the paper’s recommendation checkable through per-item outputs, derived statistics, and provenance records, with explicit limits and procedures for preserving and identifying the artifacts. It also documents maintenance terms and the status of a proposed harness integration.

I.2 Datasheet

  • 88 cell JSONL files cover every controlled-experiment model × method × seed × task; the atlas adds per-cell statistics, alongside provenance records.The atlas describes public benchmark behavior, not human-subject or personally identifying data, and represents the published, leaderboard-indexed record rather than quantization generally.Source: §I.2 · p. 75
  • Pair-task cells require binary correctness states and matching full-prompt hashes, with exclusions published alongside a reason for each row.S1 was mined from public per-item dumps, S2 from vendor evaluations, and controlled cells were generated under frozen protocols; pair enumeration was fixed before statistics were computed.Source: §I.2 · p. 75
  • 17 audited sources are not redistributed; the release provides source URLs, pinned versions, file digests, a manifest, and a retrieval script instead.Four sources lacked republication grants and seven method papers were under arXiv’s default licence. R11 and R13 have documentary rather than cryptographic provenance; the other fifteen reproduced their recorded digests on re-fetch.Source: §I.2 · pp. 75–76
  • Rev-2 is the cited revision; rev-1 remains published after an independent spot-check found a population defect despite correct arithmetic.The spot-check reconciled all 262 compared fields, and the revisions’ delta is reported.Source: §I.2 · p. 76

I.3 Metadata and identifiers

  • The paper cites the version DOI for one frozen artifact state, while the concept DOI refers to the artifact series and the superseded DOI identifies v1.0.0.The archived release and source tag share version v1.2.0; the dataset repository serves machine-readable Croissant metadata carrying the CC-BY-4.0 licence.Source: §I.3 · pp. 76–77

I.4 Maintenance

  • The author commits to at least 12 months of maintenance, with new tagged versions, changelogs, and dated registration amendments for added atlas pairs.Released versions are not edited in place, superseded revisions stay published, and the frozen pair manifest is intended to prevent silent population changes.Source: §I.4 · p. 77
  • The paired-comparison layer was proposed to lm-evaluation-harness maintainers in issue #3831, but no integration has been merged.No integration should be inferred.Source: §I.4 · p. 77

J Continuous and graded metrics: the extension this paper does not run … K.2 The procedure as executed

The appendix proposes how paired equivalence testing could extend to graded metrics, but does not execute that extension or measure the quantities it requires. It also documents that the claim audit used language-model sessions and explains what its extraction procedure can and cannot establish.

J Continuous and graded metrics: the extension this paper does not run

  • The graded-metric extension is a proposal only: no quantities were measured, and none of this paper’s required sample sizes applies beyond binary per-item outcomes.The paper’s empirical churn, score-dispersion estimates, and accuracy margin cannot be carried over to another metric; its margin must be declared and defended in that metric’s own units.Source: J Continuous and graded metrics: the… · pp. 77–78, §J.2 · p. 79

J.1 Why the binary outcome model comes first

  • Under binary correctness, the paired difference’s variance equals the probability of disagreement, enabling certification from observed per-item churn; one eligible claim was excluded because it lacked a per-item correctness state.The audit’s binary framework could not score the claim about COCO CIDEr, addressed separately in §J.4.Source: §J.1 · p. 78

J.2 TOST for a bounded graded score

  • For bounded graded scores, planning scales as (σd/m)^2, but σd must be measured for each metric, model pair, and benchmark family; binary churn and net-versus-gross counts do not transfer.A zero net difference can coexist with arbitrarily large score-change dispersion, so the graded case needs a dispersion measure rather than a count of changed items.Source: §J.2 · p. 79

J.3 How three metric families would map

  • For pass@k, k=1 is binary, while k>1 uses estimated success probabilities and requires budgets for both items and completions per item.Using the same sampling seeds on both sides can reduce estimation-error variance, but that design choice must be declared.Source: §J.3 · p. 80
  • BLEU, ROUGE, and CIDEr fit the graded framework, but corpus BLEU needs aggregation-aware certification; judge scales require accounting for judge error and defending an ordinal-scale margin.A sentence-level BLEU variant certifies a different quantity, while corpus-level certification requires paired resampling and simulation; repeated judgements are needed to estimate judge noise.Source: §J.3 · pp. 80–81

J.4 R04 and what scoring it would require

  • R04’s COCO CIDEr claim received no verdict because binary methods cannot score its graded captions, and the public record lacks paired per-caption scores, a declared CIDEr margin, and a pre-specified interpretation of negligible loss.The proposed extension does not retroactively score R04; the audit records that 0 of 16 eligible sources release task-matched per-item outputs.Source: §J.4 · p. 81

K.1 Why this appendix exists

  • The audit’s claim table was populated through language-model agent sessions, a provenance fact the appendix makes explicit.Pass 1 produced 13 rows; a fresh session produced 23 rows, then human reconciliation froze a 17-claim table before verdict or sample-size computations.Source: §K.1 · p. 82, §K.2 · p. 82

K.2 The procedure as executed

  • The fresh session prevented direct copying between extraction passes, but shared model priors could reproduce systematic misreadings, so the authors do not call the passes independent coders or report inter-rater reliability.The replacement independence mechanism was introduced after pass-1 results were known but before verdict computation; the registered inclusion and extraction rules stayed unchanged.Source: §K.2 · p. 83

K.3 Specification: what is on record and what is not … L.4.6 Dated Amendments

These parts document the preregistered procedures and record for the per-item atlas and published-claim audit, including frozen paired analyses and dated amendments to specifications, eligibility, and provenance. They describe what can be reproduced and assessed, retaining an eligible population of 16 while qualifying the candidate count and surviving power result.

K.3 Specification: what is on record and what is not

  • The extraction is auditable but not re-executable: source hashes, raw pass files, and adjudications are released, but model identifiers, prompts, and generation settings were not recorded.The author designed and froze the study, adjudicated disagreements, verified reported results, and takes responsibility for the manuscript.Source: §K.3 · pp. 83–84, §K.5 · p. 85

K.4 Agreement between the two passes

  • 11/11 jointly extracted sources agreed exactly on both registered categorical fields, while numeric discrepancies were adjudicated as differences in scope rather than value.The passes used different numeric-field scopes by design, so no quantitative numeric-field agreement rate was computed; independent human validation was not performed.Source: §K.4 · pp. 84–85

L.1.2 Experimental Grid

  • GPTQ and AWQ receive identical ordered calibration samples at each seed, so seed-level ranking differences can be attributed to method-by-calibration interaction rather than different input data.The preregistered grid includes five calibration seeds per GPTQ and AWQ model/bit cell and tests both C4 and WikiText-2 distributions on Qwen2.5-1.5B-Instruct.Source: §L.1.2 · pp. 86–87

L.1.3 Outcomes and Analysis

  • The preregistered paired analysis fixes a 2-percentage-point TOST margin for accuracy delta and specifies exact McNemar tests, confidence intervals, power-based sample sizes, and rank-flip rates.Failure to reject a difference will not be interpreted as equivalence; family-wise comparisons use Holm correction within each benchmark/model family.Source: §L.1.3 · pp. 87–88

Hierarchical aggregation across calibration seeds

  • Calibration-seed uncertainty is reported separately from item uncertainty using five seed-level accuracy estimates and a paired two-level bootstrap over seeds and shared evaluation items.Rank instability is reported both within seed and across the paired seed-by-item bootstrap, with exact-tie replicates identified separately.Source: Hierarchical aggregation across… · p. 88

L.1.4 H3 Decision Rule

  • H3 is supported if winner flips occur in at least 3 of 8 confirmatory cells or the seed-range/gap criterion holds in at least 4 of 8 cells.The confirmatory set is 4-bit GPTQ versus AWQ across four models and MMLU and GSM8K; ties count as neither flips nor non-flips.Source: §L.1.4 · pp. 88–89

L.1.6 Dated Amendments

  • 425 reconstructed WikiText-2 articles met the 2,048-token threshold against 128 required, a 3.3× eligible pool, after the registered row-level procedure found 0 of 36,718 rows eligible.The amendment defines a document as a reconstructed article and changes no other registered parameter; the builder must fail closed if fewer than 128 eligible articles remain.Source: §L.1.6 · pp. 90–91

L.2 Mini-Grid Registration: Scope, Escalation Rule, and Reporting

  • The mini-grid is frozen to execute 4 of 8 confirmatory H3 cells at 4 bits with paired C4 sets and five seeds, deferring the 7B/8B cells unless escalation is triggered.It leaves the registered H3 protocol, metrics, and decision rule unchanged; GSM8K uses test indices 0–999 and one inline few-shot example.Source: §L.2.1 · p. 92, §L.2 · p. 92, §L.2.2 · p. 92

L.2.3 3. Escalation rule (mechanical, pre-committed) / L.2.4 4. Relation to the frozen H3 decision rule

  • If escalation does not occur or the 7B/8B cells are not run, H3 is formally inconclusive on four cells, with no post hoc reduced-cell rule.The preregistered escalation checks winner flips and a range/gap criterion among four 4-bit cells; secondary analyses remain deferred.Source: §L.2.4 · p. 93, §L.2.3 · p. 93

Amendment 2 (2026-07-21, Amogh Singh): GSM8K few-shot binding

  • The GSM8K mini-grid binding is three inline few-shot examples, matching the validated bridge prompt; the specification’s literal count of one was a drafting error.The config’s `fewshot` key is boolean: a truthy value emits the fixed three-example block, rather than treating the integer as a count.Source: Amendment 2 (2026-07-21, Amogh Singh):… · p. 94

L.3 Public Per-Item Atlas Mining Registration

  • The public atlas is descriptive and exploratory, cannot test a registered hypothesis or substitute for H3, and excludes its two disclosed feasibility probes from headline aggregates.Pairs require prompt-hash identity, with cells excluded when fewer than 99% of joinable items pass; reported measures include paired flips, churn, McNemar, TOST, and required sample size.Source: §L.3 · pp. 94–95, §L.3.1 · p. 95, §L.3.4 · p. 96

L.4 Published-Claim Audit Registration / L.4.5 5. Robustness reporting / L.4.6 Dated Amendments

  • The audit assesses whether published “near-lossless” compression claims are statistically supported at their reported evaluation sizes, reporting V2’s underpowered fraction and count without judging underlying equivalence claims true or false.Robustness checks recompute verdicts using the independent-binomial bound instead of the atlas-imputed discordance rate and sweep margins of 1 pp and 3 pp; the count of margin-sensitive verdicts accompanies the headline number.Source: §L.4 · p. 97, §L.4.4 · p. 99, §L.4.5 · p. 99

L.4.6 Dated Amendments

  • The amended audit distinguishes prior numeric tolerances from result-described small deltas, applies the registered 2 pp margin as primary, and uses the 1 pp / 2 pp / 3 pp sweep for informal claims.Formal claims are additionally evaluated at their declared margin; informal claims receive no margin inferred from reported results.Source: §L.4.6 · pp. 100–101
  • Margin-dependent quantities are recomputed, while values previously based on result-derived margins are withdrawn and retained only as superseded artifact values.Recomputed quantities include V1 MDD-to-claimed-margin ratios, required-n-to-reported-n shortfall ratios, and the §5 margin-sensitivity flag.Source: §L.4.6 · pp. 99–101
  • 16 claims remain eligible after R10 was excluded because its 98.6% figure appeared only in a table, not in qualifying prose or a caption.R10 was adequately powered at the registered 2 pp margin and recorded no on V3, so its removal did not reduce the count of flagged claims or improve any count in the audit’s favor.Source: §L.4.6 · p. 101
  • The single claim below the planning threshold is a sensitivity-dependent flag: classification reverses near d = 0.1189, versus imputed d = 0.13.43.6% of the 792 atlas cells used for imputation fall below the reversal point; reports of the flag must state the imputed value, reversal point, and fraction.Source: §L.4.6 · p. 102
  • R09 and R17 stayed eligible despite §3.1’s silence; strict exclusion would shift eligible/assessable counts from 16/11 to 14/9 and the below-threshold share from 9.1% to 11.1%.Their prose comparisons gave compressed versus uncompressed scores of 73.44 versus 73.79 (R09) and 68.69 versus 68.54 (R17), without characterizing the differences; R10 had no comparative sentence and remains excluded. The permissive reading was chosen because it left published counts unchanged, while strict exclusion would have favored the audit’s thesis.Source: §L.4.6 · pp. 103–104
  • Across six quantized-model cards, three state recovery in prose, two give uncharacterized score comparisons, and one makes no comparative statement, making the 17-candidate count a floor rather than a census.The paper reports that the prose-keyed inclusion rule captures equivalence claims non-randomly.Source: §L.4.6 · pp. 103–105
  • The locus review was author re-verification against archived sources, not independent verification or dual coding.The initial eligibility correction was verdict-neutral; the rule resolving R09 and R17’s boundary case was committed before the full-text verification ran, although results were inspected before the decision.Source: §L.4.6 · pp. 102, 105
  • The full-text source tarball was removed from repository history after its presence conflicted with redistribution permissions; the 54-commit rewrite preserved the v1.0.0 tag and release archive.The corpus remains identifiable and digestcheckable through a manifest and publisher-fetch script; the amendment changed no analysis or audited property.Source: §L.4.6 · pp. 105–106
  • A second provenance remap reproduced all 54 signed commit pairs and added five mappings for commits outside the original map; the eligible population remained 16.The rewrite also revealed nine local branches and a tag referencing the tarball, plus a source-freeze pointer mismatch that was refreshed without changing file hashes.Source: §L.4.6 · pp. 107–108

Related papers

Loading 2608.15046v1…