PaperlayerView exact revision on arXiv

Source-linked AI summary

Certifying Compressed Language Models: An Audit and a Statistical Toolkit

Amogh Singh

arXiv:2608.15046v1cs.LG

TL;DR

Compressed-model equivalence claims usually rely on aggregate accuracy differences without a declared margin, test, or task-matched per-item evidence. This paper audits that gap and supplies paired equivalence testing, finding that per-item churn is roughly five times the net accuracy delta across 1,707 cells.

  • Problem

    Equivalence claims commonly lack a declared numerical margin, statistical test, and sufficient task-matched per-item evidence.

  • Method

    The paper audits 17 claims and introduces paired equivalence testing with certification tables based on observed disagreement under compression.

  • Results

    Across 1,707 paired model-by-task cells, per-item churn is roughly five times the net accuracy delta, while 16 audited claims lack task-matched outputs and advance numerical margins.

  • Takeaways & Limitations

    Checkable equivalence comparisons require a declared margin, paired testing, churn alongside net delta, sufficient sample size, and per-item outputs.

  • Takeaways & Limitations

    The atlas describes circulated evaluation evidence rather than quantization behavior in general, and the audit’s 16 sources are not a prevalence sample.

Abstract

from arXiv · show

A fraction of a point of benchmark accuracy is the usual evidence that a compressed model is equivalent to its original. That quantity is least informative when two models are most alike: a net delta is what survives cancellation between opposing per-item changes, and cancellation is most complete in the regime equivalence claims occupy. Across an atlas of 1,707 paired model-by-task cells mined from public per-item evaluation dumps (1.3B-405B), churn runs roughly five times the net accuracy delta, and cells scoring identically to their baseline still disagree on individual items. In a preregistered audit of 17 equivalence claims from three registered frames (method papers, model cards, vendor documentation), 16 are eligible. None states a prospective numerical equivalence margin, and none releases task-matched per-item outputs, though 3 release outputs for other tasks only; 5 report too little to assess numerically, so a reader cannot check them at any sample size. We audit evidential sufficiency, not truth: no claim is called false. We supply the missing instrument: paired equivalence testing at a declared margin, with certification tables giving the items an evaluation needs, computed from disagreement observed under compression, not from independent-binomial variance. A controlled experiment pairs GPTQ and AWQ on byte-identical calibration samples across five seeds. Under the frozen eight-cell decision rule H3 is supported: changing the calibration draw was sufficient to reverse the observed method ordering in 5 of 8 confirmatory cells. The reporting standard we propose is five lines: declare a margin, run the paired test, report churn beside net delta, cite the sample size you met, release per-item outputs. It applies to any comparison between two models alike enough to be worth comparing. All per-item outputs, protocols and code are released.

1 Introduction

Aggregate accuracy differences do not certify equivalence: per-item changes can cancel, even when models score identically overall. The paper audits the evidential basis of such claims and proposes paired testing with declared margins, churn reporting, adequate sample sizes, and released per-item outputs.

  • Motivation: Equivalence claims require a declared margin, a statistical test, and a sufficient sample size, but this form is almost never present in the audited sources.The relevant null is that the difference exceeds the declared margin.
  • Empirical motivation: 1,707 paired model-by-task cells show per-item churn at roughly five times the net accuracy delta, including cells with baseline-matched accuracy but item-level disagreements.The atlas spans public per-item evaluation dumps for models from 1.3B to 405B parameters.
  • Audit: 16 of 17 audited equivalence claims were eligible; none declared a numerical tolerance or released task-matched per-item outputs, and 5 could not be assessed numerically.Three sampling frames were method papers, official quantized model cards, and inference-stack vendor documentation; no claim was labeled false.
  • Statistical toolkit: Certification tables compute required item counts from observed paired disagreement under compression rather than independent-binomial variance, with requirements set by churn.The tables target certification within ±m points of a baseline.
  • Reporting standard: The proposed five-line reporting standard declares a margin, runs a paired test, reports churn beside net delta, cites the met sample size, and releases per-item outputs.The standard is intended for comparisons between models alike enough to be worth comparing.

2 Related work and positioning

Prior work established aggregate-accuracy masking, calibration-data effects, and difference detection, while this paper positions its contribution as large-scale decomposition, paired equivalence certification, and auditing reporting sufficiency. Its standard addresses whether alike models are equivalent, not merely whether they differ.

  • Positioning against prior work: Dutta et al. show compressed models can match baseline aggregate accuracy while flipping individual answers; this paper adds large-scale decomposition, flip-rate variance modeling, and an ensuing audit.The paper explicitly attributes the flips metric to Dutta et al. and does not claim it.
  • Reporting audit: Across 16 eligible audited sources, 0 declare a prospective numerical margin and 0 release task-matched per-item outputs.The paper distinguishes reporting insufficiency from a claim that the audited equivalence statements are false.
  • Calibration effects: Williams and Aletras establish substantial downstream-performance variation across calibration sets; this paper instead holds the corpus fixed and tests whether calibration-sample seeds change method rankings.The authors report that their sweep found no prior work asking this seed-level ranking question with the calibration corpus held fixed.
  • Equivalence versus difference detection: Kübler et al. apply one-sided McNemar detection to LLM accuracy comparisons, whereas this paper uses TOST certification at a declared margin with required-n tables.The paper argues that non-significance is not equivalence and may reflect insufficient resolution at the sample sizes considered.

3 Paired certification: what an equivalence claim has to show

Paired equivalence certification requires a declared margin and a TOST-style sample-size plan based on item-level discordance, not a non-significant difference or independent-binomial variance. The resulting counts are design lower bounds: they certify equivalence only when the evaluation meets the required sample size and reports the paired result accordingly.

  • Why paired equivalence testing: A non-significant McNemar result does not establish equivalence, because an underpowered evaluation may simply fail to detect a difference.The framework distinguishes absence of evidence for difference from evidence that performance lies within a declared margin.
  • Limits of the design tables: Every reported nreq is a lower bound when the true difference is nonzero, because TOST must resolve m −|δ| rather than the full margin m.The requirement grows without bound as |δ| approaches m, so the tables are designs rather than retrospective certifications.
  • Planning sample sizes: 2,164 items are required for MMLU at a 2 pp margin under typical discordance, versus 7,727 when pairing is ignored.The paired calculation uses item-level disagreement between the two models rather than independent-sample variance.
  • Planning sample sizes: MMLU requires 2,164 items while GPQA requires 749 at 2 pp, because typical discordance is 0.140 versus 0.048, not because of benchmark difficulty.GPQA’s median baseline accuracy is 0.373, compared with MMLU’s 0.492.
  • Scope and reporting: Binary per-item correctness is required; benchmarks without measured discordance rows must estimate discordance on the practitioner’s own model pair before sizing the evaluation.Graded metrics, judge ratings, pass@k for k > 1, and perplexity-style measures require an unvalidated continuous adaptation.
  • Scope and reporting: If the evaluation falls short, practitioners must add items, widen the declared margin, or report the paired difference and interval without calling it certified.Reporting a non-significant difference as equivalence is explicitly excluded.

4 The atlas: cancellation at scale, and what certification costs

Across 1,707 paired model-by-task cells, per-item churn substantially exceeds net accuracy change, including cells whose aggregate accuracy is exactly unchanged. Certification is costly and often inconclusive at observed sample sizes, while the atlas remains bounded by its public-record population and cannot establish causal claims.

  • Atlas population: 1,707 cells formed the analysis population, spanning 1.3B–405B models after exclusions and feasibility probes were removed.The population comprised S1 = 1,398 and S2 = 309 cells; empty joins can make aggregate differences unpaired and preclude per-item analysis.
  • Cancellation at scale: 5.40 was the pooled ratio of median accuracy-state churn to median absolute accuracy change, with 5.22 in S1 and 5.19 in S2.The ratio was nearly unchanged despite median churn and net delta each falling by roughly a factor of 2.86–2.85 from S1 to S2.
  • Certification cost: 69.2% of S1 cells—967 cells—fell into a gray zone: neither equivalent at the registered 2 pp margin nor detectably degraded.Only 4.9% were certifiably equivalent and 26.5% showed a detectable difference; smaller margins require more items.
  • Cancellation at scale: 145 of 1,707 cells (8.49%) matched baseline accuracy exactly, yet 128 had nonzero churn.Across these 145 cells, median churn was 0.0720, mean churn 0.0887, and maximum churn 0.3434.
  • Scope and limitations: The atlas supports claims only for its measured population and cannot establish general quantization effects or causal attribution.Its record is conditioned on leaderboard coverage, inherits original evaluation item counts, and motivated a controlled experiment because it cannot answer causal questions.

5 What published equivalence claims actually report

The audit found that published equivalence claims generally lack a prospective numerical margin and task-matched per-item outputs, limiting independent assessment. Of 16 eligible claims, 5 were not numerically assessable, while the assessable evaluations were largely sized for a registered 2 pp margin.

  • Reporting evidence: 0 audited sources declare a prospective numerical equivalence margin; 10 use qualitative language, while 6 report only post-evaluation outcomes.Without a declared margin, a claim cannot be evaluated against its own standard and must instead be assessed against an external registered margin.
  • Reporting evidence: 0 of 16 eligible sources release task-matched per-item outputs for the tasks underlying their equivalence claims.Three sources release per-item outputs for other suites, but those artifacts do not support the audited claims.
  • Numerical assessment: Of 11 assessable claims, 1 fell below the approximate planning threshold at the registered 2 pp margin under median discordance imputation, but its classification did not survive sensitivity analysis.The flagged claim, R01, reported n = 1,838 against a requirement of 2,010 items at an imputed discordance rate of 0.130.
  • Numerical assessment: 10 of 11 assessable evaluations were sized to carry their claims at the registered 2 pp margin, while no claim fell below the threshold throughout the sensitivity interval.The result concerns evidential resolution rather than truth or error: no audited claim was described as false.
  • Numerical assessment: 5 of 16 eligible claims cannot be assessed numerically, including 4 with insufficient reporting and 1 outside the registered binary paired-outcome model.Missing inputs included sample size, baseline, and extractable numerical evidence; R04 reports COCO CIDEr, which lacks a per-item correct/incorrect state.

6 Cancellation is worse where practitioners choose

In the controlled GPTQ–AWQ comparison, calibration changes were sufficient to reverse the observed winner in 5 of 8 cells, while cancellation made per-item disagreement much larger than aggregate accuracy differences. The evidence was stronger on MMLU than GSM8K, where n = 1,000 often lacked resolving power as predicted by the certification table.

  • Cancellation: 12.7 was the median per-cell churn-to-net-delta ratio across eight controlled cells, versus 3.85 across the 1,707-cell atlas.The controlled method-against-method contrast showed the larger ratio under both aggregations.
  • Cancellation: 0.58 pp separated GPTQ and AWQ on Qwen2.5-7B/GSM8K, while 17.7% of items changed correctness state and 28.7% of answers changed outright.The passage identifies cancellation as the mechanism behind the small aggregate gap despite substantial item-level changes.
  • Task dependence: All four MMLU cells met the range/gap criterion, with seed-induced ranges of 5.5 to 17.5 paired standard errors; GSM8K ranges were roughly two to three-and-a-half SE at n = 1,000.On MMLU, seed-level SD exceeded item-level SE by 1.6 to 5.9×, whereas GSM8K’s components were of the same order.
  • Certification resolution: 1,184 items were required at median discordance and 3,068 at p75 discordance for GSM8K at a 2 pp margin, exceeding the confirmatory n = 1,000.Observed discordances were pd = 0.1766, 0.1808, 0.2086 and 0.2940, with requirements near three times the sample used at or above p75.

7 Exploratory scoring-pipeline sensitivity

On a fixed FP16 model, harness configuration changed per-item correctness by an amount comparable to quantization, sometimes exceeding it. The exploratory study therefore shows that unstated or mismatched evaluation settings can dominate compression comparisons.

  • The study was small, preregistered, exploratory, and conducted on one model with matched bridge-item subsets.Its question was how harness configuration affects accuracy and per-item answers relative to quantization under a fixed configuration.
  • ¯Q(MMLU) = 0.199 and ¯Q(GSM8K) = 0.287 were the mean churn denominators for ten quantized variants.These variants were compared against the FP16 cell on the same items.
  • R ∈[0.836, 1.585] across five conditions, showing configuration changes can move per-item correctness as much as or more than quantization.R compares configuration-induced churn with mean quantization churn on the same items.
  • R = 1.585 for the scoring-filter choice exceeded the quantization denominator outright.The protocol reports both inputs to the ratio in every Table 20 row.

8 Artifacts

The paper releases six artifacts, including per-item outputs for all 88 controlled-experiment cells, alongside evaluation, audit, and certification materials. The canonical frozen release is v1.2.0, with reproducibility metadata provided for audit-source captures that cannot be redistributed.

  • Released artifacts: 88 per-item JSONL files cover every controlled-experiment cell, addressing the audit’s finding that 0 of 16 eligible sources released task-matched outputs.The release also includes the flip atlas, flipeval package, frozen 17-claim audit table and verdict CSV, and certification materials.
  • Audit-source materials: Audit-source full-text captures remain private when redistribution rights are unavailable, but the release provides each source’s URL, pinned version identifier, SHA-256 hash, manifest, and retrieval script.The archive includes four sources without grants permitting third-party republication and seven arXiv papers whose default licence authorises arXiv rather than the authors.
  • Canonical release: The canonical archived release is Zenodo 10.5281/zenodo.21939143, corresponding to frozen v1.2.0, with the package tagged v1.2.0 on GitHub and data mirrored on Hugging Face.Version 1.0.0 is historical and predates the rev-3 audit verdicts.

9 Limitations

The evidence is bounded by a nonrepresentative atlas, fixed audit frames, and a narrowly registered binary experiment. Results do not extend to untested settings or open-ended, float-scored generation, although paired analysis can handle graded metrics with metric-specific variance measurement.

  • Evidence coverage: The atlas describes circulated evidence, not quantization behaviour in general, because it covers community quantizations conditioned on leaderboard coverage and one vendor’s releases.Its pipeline also had two bugs found and fixed during the run.
  • Statistical aggregation: Certification-family quartiles are conservative, and two families with fewer than 12 cells are indicative only.Per-subject and per-subtask cells are collapsed into families, mixing subject-level with model-level variation.
  • Audit scope: Audit verdicts apply per claim at the pooled frozen-table sample size, and only claims from the three registered frames are audited.Claim × benchmark analysis would require inventing rows absent from the freeze; later-discovered claims would form a separate population.
  • Experiment scope: The controlled experiment covers only 8 registered confirmatory cells spanning 1.5B to 8B models, two benchmarks, 4 bits, two methods, and five seeds.The verdict therefore licenses no statement about 3-bit behaviour, ARC-Challenge, HellaSwag, or other excluded settings.
  • Metric scope: English, predominantly multiple-choice or short-answer benchmarks and binary per-item correctness exclude open-ended generation and 33 float-scored atlas cells.The CIDEr-style metric that placed R04 outside the registered binary paired-outcome calculation is excluded; this limits the registered flip model, not paired analysis generally.
  • Generalisation: For graded metrics, paired testing requires measuring per-metric, per-family variance and declaring margins in the metric’s own units.Binary correctness has di ∈ {−1, 0, +1} and variance under a true zero difference equal to the discordance rate pd; the paper measured no graded-metric variance.

10 Conclusion: a reporting standard for compression claims

The conclusion argues that aggregate accuracy cannot certify equivalence among similar models because item-level churn can hide behind small net deltas. It therefore calls for checkable comparisons grounded in declared margins and per-item evidence, including comparisons beyond compressed models versus originals.

  • Why aggregate accuracy fails: 0.58 points of GSM8K separation concealed disagreement on 17.7% of items between two 4-bit Qwen2.5-7B quantizations.Certifying equivalence at a declared ±2-point margin required 2,730 items, while the evaluation used 1,000.
  • Why aggregate accuracy fails: Median churn was about five times the median net accuracy delta across 1,707 public model-by-task cells.The conclusion states that aggregate accuracy is not a summary of behavior or of what compression changed.
  • Reporting evidence: Across 16 eligible audited sources, none declared a prospective numerical equivalence margin or released task-matched per-item outputs.3 released outputs for other tasks only, and 5 could not be assessed from their reports; these are audited-sample counts, not literature estimates.
  • Scope of the standard: The proposed standard applies to any comparison between models similar enough to compare, not only a compressed model against its original.The conclusion explicitly extends it to method, checkpoint, and version comparisons, where per-item evidence and a declared margin make claims checkable.

A Reconciling the calibration-sensitivity antecedents … B.4 The H3 reporting rule, stated before the results exist

The paper reconciles conflicting calibration-sensitivity findings by distinguishing absolute accuracy from method-ranking stability, then preregisters a frozen, mechanically applied eight-cell rule and documents the audit’s reporting safeguards. It situates the work as a constructive audit spanning compression families, losslessness definitions, related measurements, and preregistration practice.

  • A Reconciling the calibration-sensitivity antecedents: Williams and Aletras find substantial calibration-set variation, whereas Paglieri et al. find modern models robust, including Mistral 7B as nearly immune.The paper treats both as careful studies reaching opposite conclusions about calibration sensitivity.
  • A Reconciling the calibration-sensitivity antecedents: Holding the corpus fixed while varying only the sample seed targets a finer intervention than changing calibration-set quality, content, or language.The paper also compares method gaps at matched bit widths rather than studying one method at a time.
  • A Reconciling the calibration-sensitivity antecedents: In 7 of 8 confirmatory cells, the seed-induced range was at least as large as the mean GPTQ–AWQ gap, implying ranking instability despite individual robustness.The paper argues that absolute stability can coexist with unstable relative ordering.
  • A.1 Adjacent concurrent measurements: Cacioli recommends reporting churn beside the mean, while Nikolić et al. find KL-divergence proxies lose ranking signal near the baseline.These adjacent measurements overlap with the paper’s focus on per-item comparisons and near-baseline certification.
  • A.2 Losslessness definitions and fidelity metrics: Helcig et al. define task-lossless and distribution-lossless quantization and propose Expected Acceptance Rate, whereas this paper audits evidence and computes certification sample sizes.Their paper contains no equivalence test at a stated margin.
  • A.3 The constructive-audit genre, and preregistration precedent: Following constructive-audit precedents, the paper evaluates what the field reports rather than whether individual authors are correct, delivering a standard plus tooling.The framing also follows emerging preregistration practice.
  • A.4 The compression families the audited claims span: The audit spans weight-only, weight-and-activation, and one-shot-pruning families, including GPTQ, AWQ, SqueezeLLM, LLM.int8(), SmoothQuant, SparseGPT, and Wanda.It audits equivalence language and supporting evidence, not the methods themselves.

B.5 Disclosed pre-registration data contact … D.4 Discordance imputation

The paper discloses preregistration contacts, corrects protocol and population errors, and publishes revisions and audit data so equivalence results remain inspectable. It also defines paired-test verdict quantities and imputes discordance from matched atlas cells, while exposing limits in claim placement and match specificity.

  • B.5 Disclosed pre-registration data contact: 74 flips of 1,170 (net −1.03 pp) and 17 flips of 250 (net +1.2 pp) came from preregistration feasibility probes excluded from headline aggregates.The exclusion covers all 99 cells belonging to the two probe pairs.
  • B.6 The interpretive choices that moved the headline: The analysis sequence changed from K = 1 of 12 to K = 5, then K = 4, before Amendment 2 restored the registered margin and yielded 1 of 11 assessable claims below threshold.The superseded implementation incorrectly substituted reported deltas for prospective margins; Amendment 2 applies the uniform registered 2 pp margin.
  • B.6 The interpretive choices that moved the headline: R04 is indeterminate rather than scored on a substituted benchmark, while paired variance and median-discordance imputation remain the analysis’s resolved choices.Its retained GSM8K computation appears only as a labelled transparency column in the released CSV.
  • B.7 Atlas validation and population correction: All 262 compared fields reconciled exactly, but validation found the population wrong because the implementation omitted the registered reverse-chronological fallback after prompt-hash failure.Eleven pairs contributed zero cells, and the loss was systematically associated with later reevaluation of quantized models.
  • B.8 The rev-1 to rev-2 delta: The repair recovered cells rather than removing them: analysed cells rose from 1,254 to 1,807, and probe-excluded analysable cells from 1,155 to 1,707.The enumerated population stayed at 2,055 pair-task cells; both revisions and their delta are public.
  • B.8 The rev-1 to rev-2 delta: S1 median churn rose from 0.1327 to 0.1375, S1 median |net delta| from 0.0226 to 0.0263, and certifiability at 2 pp fell from 5.6% to 4.9%.The detectable-difference share rose from 17.5% to 26.5%, while every S2 field remained identical.
  • C Full audit table: The released CSV records each claim’s identity, evidence basis, sample-size basis, imputed discordance, assessability, reproducibility, variance-model quantities, and superseded readings.Transparency columns retain computable quantities for ineligible or non-assessable claims without treating them as verdicts.
  • D.1 Where the equivalence claim is written: Six cards shared identical benchmark evidence, no declared margin, and no task-matched per-item outputs, yet differed in where their equivalence claims appeared.The audit retained two prose-juxtaposition cards because near-identical scores functioned as equivalence assertions despite lacking trigger vocabulary.

D.5 The two robustness readings … E Certification tables at 1 pp and 3 pp

The audit fixes the applicable margin at 2 pp, excludes one ineligible claim, and finds that remaining claims often lack the reporting needed for numerical assessment. Robustness checks and certification tables show that conclusions depend on the margin, imputed discordance, pairing assumptions, and sample size.

  • D.5 The two robustness readings: The registered 2 pp margin now applies uniformly; the previously claimed factor-of-four interpretive gap is withdrawn.Margin sensitivity flips R01 across 1 pp →3 pp, while imputation sensitivity reverses at d = 0.118915.
  • D.6 Eligibility, and the full-text source review: The eligible denominator moves from 17 to 16 after full-text review excludes R10 under the registered inclusion rule.The review searched complete sources, including tables, captions, footnotes, appendices, reference lists, and raw Markdown model cards.
  • D.7 The complete-text margin sweep: None of the 17 archived sources declares a prospective numerical equivalence margin.The complete-text sweep found no occurrences of parity or percentage point and no qualifying use of margin.
  • D.8 Per-item outputs: what the three partial sources release: Three Red Hat AI model cards release per-item outputs for other tasks, but none for the OpenLLM v1 tasks underlying their audited claims.Without task-matched outputs, outsiders cannot run paired tests, compute churn, or check arithmetic at any sample size.
  • D.9 The non-assessable claims, one by one: Four claims are non-assessable for insufficient reporting, while one lies outside the registered calculation.R02 and R11 omit sample size, baseline, and numeric delta; R13 gives n = 250 without a baseline; R14 reports 0.7 pp without a baseline; R04 concerns an unscorable quantity.
  • D.10 R01: the planning values behind the single flag: R01 reports n = 1,838 against a requirement of 2,010 items at imputed discordance 0.130, with adequacy reversing at discordance 0.1189.Across the interquartile range, the requirement spans 1,364 to 4,328 items; R01 is adequately sized at the first quartile and undersized at the third.
  • D.11 R04: the overruled first-pass computation; D.12 Interpretive rulings and superseded verdicts: The GSM8K result for R04 (−0.30 pp at n = 1,319) is overruled because it audited a different benchmark and language.Amendment 2 makes the uniform 2 pp margin primary and forbids treating source-reported quantities as declared margins.
  • D.13 Both robustness directions in full; E Certification tables at 1 pp and 3 pp: Under pairing, only R01 is coarser than the registered margin, while ten evaluations resolve differences between one-sixth and one-half of it.Independent-binomial columns are roughly twice as demanding; MMLU requires 8,656 items at 1 pp, 2,164 at 2 pp, and 962 at 3 pp.

E.1 Certification method detail … H.5 Why MMLU collapses to a single off-reference cell

The paper provides a paired certification toolkit grounded in observed churn, audits atlas construction and scope, and examines controlled calibration, harness, and configuration sensitivity. Its evidence emphasizes that net accuracy differences can conceal substantial item-level disagreement and methodological instability.

  • E.1 Certification method detail: Paired variance uses discordance pd, not independent-binomial accuracy variance, because the same items are evaluated by nearly identical models that agree on most items.The paired and independent columns therefore differ in their variance assumptions; low-churn families benefit most from pairing, while MATH and MMLU require more items and gain less.
  • E.2 Scope caveats carried from the main text: 2 families remain thinly evidenced: mmlu_pro has 5 cells and ifeval has 8, so their quartiles are hypotheses to replace with measured churn.MMLU has 1,311 cells, BBH 192, and MATH 56; aggregation across subjects and subtasks makes quartile bands conservative for single-model evaluation.
  • F.3 Exclusion breakdown; F.4 The most extreme zero-delta cell: 99 feasibility-probe cells were excluded because hand-built pairs had sample sizes as low as n = 10 and discordance rates as high as 0.9.Including them would distort the quartiles used by the certification tables.
  • F Atlas construction detail; F.1 The two sources; F.2 Item pairing; F.3 Exclusion breakdown: 2,055 pair-task cells were enumerated, 1,807 analysed, and 248 excluded or skipped for missing files, empty joins, or absent binary correctness metrics.The atlas draws from the Open LLM Leaderboard archive and Neural Magic/Red Hat dumps; pairing requires identical item keys and full-prompt hashes, with a 99% identity threshold.
  • G Controlled experiment: supporting detail; G.1 The mechanical escalation screen: 2 of 4 mini-grid cells showed winner flips and 3 of 4 met the range/gap criterion, triggering construction of deferred 7B/8B confirmatory cells.The screen is distinct from the eventual eight-cell H3 verdict and was run on sealed cells after 409/409 validation checks.
  • G.2 Per-seed differences behind the winner-flip column; G.3 The three registered supporting analyses; G.4 Post-hoc resolution analysis; G.5 Deferred registered analyses: 5 cells had winner flips across calibration seeds, with no exact ties, while registered supporting analyses used 2,000 bootstrap replicates at RNG seed 0.The post-hoc resolution analysis was unregistered, tested no hypothesis, and did not modify the verdict; several registered secondary analyses remained deferred.
  • G.6 The earlier public-checkpoint pilot: −4.25 pp at n = 400 was observed for public GPTQ on MMLU with exact McNemar p = 0.036, but the pilot remains exploratory evidence only.Its protocol used raw-text prompts, an unpinned modified image, and fixed or undocumented calibration, so it cannot test H3.
  • H Harness-sensitivity detail; H.1 Two observed defects; H.2 Design and the pre-named ratio; H.3 Condition B in full; H.4 The same move, in another domain; H.5 Why MMLU collapses to a single off-reference cell: 0 GPU time was required for Condition B, whose identical-generation rescore produced correctness churn CB = 0.455 (91 of 200 items).The harness study found configuration effects, defined R = Ccond/¯Q with ¯Q(MMLU) = 0.199 and ¯Q(GSM8K) = 0.287, and collapsed MMLU’s conditions C and D as byte-identical.

H.6 Condition-by-condition results · I Artifact detail · I.1 The six released artifacts

Condition changes can match or exceed quantization-induced movement, while the released artifacts provide item-level data, analysis tools, manifests, and audit materials subject to redistribution limits. The results show why net accuracy alone is insufficient for paired model comparisons.

  • H.6 Condition-by-condition results: Three of four GSM8K conditions reached R ≥1, matching or exceeding the per-item movement from swapping in one of ten quantized variants.Only multiturn exemplar placement remained below the quantization scale; net deltas understate movement because they cancel directional churn.
  • H.6 Condition-by-condition results: +4.5 pp on MMLU followed turning the chat template off, changing accuracy from 0.415 →0.460 with a directional split of 33/51.The associated churn was 0.210 (84 of 400 items), giving R = 1.055; the harness default was better on MMLU but worse on GSM8K.
  • I.1 The six released artifacts: 88 cell JSONL files provide one row per evaluation item across every model × method × seed × task in the controlled experiment.Each row carries the per-item correctness state consumed by the paired test.
  • I.1 The six released artifacts: Two sealed run archives preserve the same bytes as the extracted files, with per-file SHA-256 and archive checksums recorded at run completion.The checksums allow the loadable copies to be checked against the runtime record rather than trusted.
  • I.1 The six released artifacts: The flip atlas supplies paired statistics for every enumerated pair-task cell, alongside the exclusion table and frozen 59-pair manifest.The package includes flip rates, churn measures, exact McNemar, TOST, bootstrap intervals, rank-flip rates, minimum detectable difference, required-n, and certification-table generation.
  • I.1 The six released artifacts: The audit artifacts include a frozen 17-claim table with source content, while flipeval is released under Apache-2.0 and other materials under CC-BY-4.0.The supplied passage identifies these as part of the released audit and analysis materials.
  • I.1 The six released artifacts: Four audited sources lacked grants permitting third-party republication, and seven method papers were distributed under arXiv’s default licence without authorising republication by the authors.The redistribution review therefore concluded that publishing the corpus was never an option.

I.2 Datasheet · I.3 Metadata and identifiers · I.4 Maintenance

The released artifacts make the paper’s recommendation checkable through per-item outputs, derived statistics, provenance records, and auditable population and revision controls. Canonical identifiers, licensing constraints, and maintenance procedures preserve reproducibility without implying unmerged external integration.

  • I.2 Datasheet: 88 cell JSONL files provide one per-item row for every model × method × seed × task in the controlled experiment, alongside statistics and provenance records.The statistics include atlas flip and churn quantities plus McNemar and TOST outcomes; provenance includes registrations, signed decisions, configurations, receipts, fingerprints, and an incident log.
  • I.2 Datasheet: S1 cells come from public evaluation dumps, S2 from vendor evaluations, and controlled cells from pinned-container runs under pre-frozen protocols and pair enumeration.The pair manifest was frozen before any statistic was computed.
  • I.2 Datasheet: A pair-task cell is admitted only when both models have binary per-item correctness states and matching full-prompt hashes; exclusions are released with row-level reasons.This makes the admitted population auditable rather than merely asserted.
  • I.2 Datasheet: The artifacts are CC-BY-4.0, while flipeval is Apache-2.0; because S1 has no declared license, only derived statistics are redistributed and source identifiers enable re-derivation.Site-specific identifiers were replaced with placeholders without altering accuracy, hashes, counts, seeds, timestamps, job IDs, or decision values.
  • I.2 Datasheet: Rev-2 is cited while rev-1 remains published: an independent spot-check reconciled all 262 compared fields, finding correct arithmetic but a defective population.Publishing both revisions keeps the correction visible rather than overwriting the original artifact.
  • I.3 Metadata and identifiers: Version v1.2.0 is shared by the source tag, archived release, and dataset revision, with DOI 10.5281/zenodo.21939143 canonical for citation.The concept DOI identifies the artifact series, while the superseded v1.0.0 DOI helps readers identify an earlier release.
  • I.4 Maintenance: The author will maintain the release for at least 12 months, target a two-week first response, and publish fixes as new tagged versions with changelogs.Released versions are not edited in place, superseded revisions remain available, and atlas additions require dated registration amendments.
  • I.4 Maintenance: The paired-comparison layer was proposed to lm-evaluation-harness maintainers in issue #3831, but no integration has been merged.No external integration should be inferred from the proposal.

J Continuous and graded metrics: the extension this paper does not run … K.2 The procedure as executed

The paper’s certification framework is limited to binary per-item correctness: graded metrics require new variance, margin, and aggregation decisions, and this extension is proposed but not executed. The audit procedure used separate language-model sessions plus human reconciliation, while acknowledging that shared model priors can preserve systematic extraction errors.

  • J Continuous and graded metrics: the extension this paper does not run: No part of the continuous-metric extension was executed, and none of this paper’s sample-size requirements applies beyond the binary per-item model.The extension is explicitly a derivation and proposal rather than an empirical result.
  • J.1 Why the binary outcome model comes first: Binary correctness supplies an exact paired variance because each per-item difference is −1, 0, or +1, making its second moment the disagreement probability.This identity lets certification tables use measured per-item disagreement rather than an assumed joint distribution.
  • J.2 TOST for a bounded graded score: For bounded graded scores, paired TOST retains the same form but replaces binary disagreement with the measured standard deviation of per-item score differences.The planning requirement scales as (σd/m)^2, with the binary case recovered when σd^2 = pd.
  • J.2 TOST for a bounded graded score: Graded metrics do not inherit binary churn or the registered 2 pp margin: dispersion must be measured, and margins must be declared in each metric’s own units.The paper has no path from its accuracy-state atlas to σd for CIDEr, ROUGE, or judge scales.
  • J.3 How three metric families would map: For pass@k with k > 1, evaluation requires separate budgets for items and completions per item because estimated success probabilities add sampling error to graded differences.Shared sampling seeds can positively correlate the two sides’ estimation errors, but that design choice must be declared.
  • J.3 How three metric families would map: BLEU, ROUGE, and CIDEr are graded per item, but corpus-level aggregation is not the mean of per-item scores, so Equation (6) applies only after σd is measured.The aggregation obstacle is distinct from the graded-score variance calculation.
  • J.4 R04 and what scoring it would require: R04 received no verdict because COCO CIDEr lacks per-item correctness states; scoring it would require paired caption scores, a defended CIDEr margin, and a prespecified claim interpretation.Equation (6) does not retroactively supply those missing inputs.
  • K.2 The procedure as executed: The audit claim table was assembled by language-model agent sessions, then human reconciliation merged and froze the records before verdict or power calculations.Separate sessions prevented direct leakage but did not provide independent coders: shared model priors could reproduce systematic misreadings in both passes.

K.3 Specification: what is on record and what is not … L.4.6 Dated Amendments

The paper preregisters a reproducible framework for paired equivalence, churn, calibration-seed instability, and published-claim auditing, while documenting limits on re-execution and later protocol amendments. Its amendments preserve the published audit quantities while clarifying sampling, eligibility, margins, and repository handling.

  • K.3 Specification: what is on record and what is not: The extraction is auditable but not re-executable because model IDs, prompts, sampling settings, and retry policies were not recorded or released.Readers can verify extracted fields against source hashes and raw pass files, but cannot reproduce the extraction run itself.
  • K.4 Agreement between the two passes: 11/11 jointly extracted sources agreed exactly on both categorical fields: per-item outputs released and statistical test or interval reported.Numeric discrepancies reflected extraction scope rather than value, and no independent human validation was performed.
  • L.1.1 Claims: The preregistered framework distinguishes net accuracy delta from gross churn, treats calibration seeds as random effects, and reports paired bootstrap uncertainty and rank-flip rates.TOST uses a fixed 2 percentage-point accuracy-delta margin, while failure to reject a difference is not interpreted as equivalence.
  • L.1.2 Experimental Grid: The frozen protocol specifies paired execution, fixed benchmark item sets, logged exclusions, and identical prompts and scoring across compared methods.The grid includes GPTQ and AWQ calibration seeds {0, 1, 2, 3, 4}, while failures are rerun with the same registered seed and calibration indices.
  • L.1.4 H3 Decision Rule: The primary H3 test compares 4-bit GPTQ and AWQ across 8 model-by-benchmark cells, supporting H3 when winner flips occur in at least 3 cells or the range/gap criterion holds in at least 4.Winner flips require opposite nonzero method gaps across registered seeds; ties are reported separately, and 3-bit results are secondary.
  • L.1.6 Dated Amendments: The WikiText-2 amendment defines a document as a deterministically reconstructed article after a pre-decision failure found 0 of 36,718 rows eligible under the original row-based interpretation.The eligibility probe found 425 eligible articles among 629 reconstructed, exceeding the requirement of 128; registered seeds, sample count, token length, pairing, and index retention remained unchanged.
  • L.4.4 4. Verdict rules (mechanical, computed only after §3.4 freeze): The audit reports evidential sufficiency rather than truth, and later repository and locus amendments left every published count unchanged.Three of six quantized-model cards lacked trigger vocabulary in prose, making the frozen candidate count of 17 a floor rather than a census; the source archive also conflicted with redistribution review.
  • L.4.6 Dated Amendments: The audit amendments corrected margin handling and R10 eligibility while retaining boundary cases R09 and R17, leaving the eligible population at 16 and the assessable population at 11.The corrected margin recomputed margin-dependent quantities, while R09 and R17 remained eligible under the permissive reading despite prose score differences of 0.35 pp and 0.15 pp.
Loading 2608.15046v1…