PaperlayerView exact revision on arXiv

Source-linked AI summary

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints

Haoyaun Zhu, Jie Zhang

arXiv:2609.04198v1cs.AIcs.LG

TL;DR

Black-box LLM judges are increasingly used as measurement instruments, but the stability of a fixed request and model name is rarely validated. This paper audits that assumption with two preregistered campaigns and finds that both fail frozen reliability gates despite perfect execution records, motivating instrument-first evaluation before gate-setting.

  • Problem

    LLM judges rank responses, score traces, gate training data, and drive leaderboards, yet the stability of their measurements on shared endpoints is rarely established.

  • Method

    The paper preregisters and audits an external two-model measurement pipeline, testing repeat rankings and byte-identical replays before scientific claims proceed.

  • Results

    52,988 audited request attempts failed both frozen instrument gates: same-window Spearman agreement was 0.400 versus 0.90, and next-day replay agreement was 0.78 versus 0.99.

  • Takeaways & Limitations

    On shared endpoints, a model name is not a frozen instrument, so evaluations should measure instrument status before freezing downstream gates.

  • Takeaways & Limitations

    The conclusions are bounded by the tested configurations and externally measured visible behaviour, not model internals or universal provider quality.

Abstract

from arXiv · show

Language-model judges now gate training data, score generations, and drive leaderboards. The judge is then a measurement instrument, resting on one rarely stated assumption: the same request, sent to the same model name, reads the same tomorrow. We audited that assumption in two preregistered campaigns with every threshold fixed in advance; neither got past validating its instrument. Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99, each time with the execution record at ceiling. Three mechanisms explain the gap: a label-to-meaning mapping that biased readouts as strongly as the signal; candidate gaps seven orders of magnitude below the instrument's own noise floor; and byte-identical inputs returning different rankings, a noise that exact-permutation readouts compound. Neither metric substitution nor sampling repaired it on the tested grid. Preregistered follow-ups bound the problem: waiting did not help on the days sampled (0.805 versus 0.800, replicated over five further days); switching providers did not help (four providers share the floor, medians 0.74 to 0.88, predicted by none of the metadata fields they expose); self-hosting on batch-invariant kernels helped only while the server was quiet; and on constructed errors with known gaps, the readout's separation tracks error type, not size. We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist; a pilot at roughly 2% of the study's call volume would have exposed both unreachable gates in advance. All results concern externally measured behaviour on shared serving infrastructure. On a shared endpoint, a model name is not a frozen instrument; a preregistered evaluation must measure its instrument before freezing any gate on it.

1 Introduction

This paper audits whether black-box LLM judges are stable measurement instruments on shared endpoints. Two preregistered campaigns failed instrument-validation gates despite ceiling-level execution records, motivating an instrument-first evaluation discipline.

  • Audit result: 52,988 audited request attempts produced same-window repeat-ranking agreement of Spearman 0.400 versus a frozen 0.90 gate and next-day replay agreement of 0.78 versus 0.99.Both campaigns terminated at the instrument layer while requests, schemas, hashes, and recorded metadata remained valid and constant.
  • Mechanisms: Three mechanisms—mapping bias, near-degenerate candidate separations, and platform nondeterminism—explain the instability and are amplified by permutation readouts.Metric substitution and sample-size scaling did not repair the problem on the tested grid.
  • Attribution: The paper attributes instability through same-day and cross-day variance decomposition, four-provider comparisons, self-hosted tests, and constructed-error supplements.The four-provider characterization found comparable floors, while system_fingerprint did not predict stability in three failure modes.
  • Design implications: The study contributes an L0–L2 snapshot-identity ladder and makes snapshot identity mandatory metadata alongside temperature and seed.The ladder is paired with a taxonomy of boundary forms observed during the audit.
  • Design implications: An instrument-first pilot–simulate–freeze pipeline uses operating-characteristic calibration before evaluation gates are frozen.The study estimates that a pilot of about a thousand calls could have redirected the roughly fifty-three-thousand-attempt sequence.
  • Scope: The claims concern externally measured behaviour on tested shared-serving configurations, not model internals or provider quality.The paper explicitly separates visible-record measurements from a separate white-box theory-testing track.

2 Background and related work

Prior work documents judge bias, self-inconsistency, serving nondeterminism, and reproducibility practices, but does not combine them into a preregistered instrument audit with enforced stability gates. This paper positions its contribution as that assembled whole and the discipline derived from it.

  • Judge reliability: Judge research has catalogued position, verbosity, self-enhancement, and self-preference biases, while repeated-call studies report generally low self-consistency.Earlier reliability values inform later studies but do not bind the study that measures them.
  • Nondeterminism: Serving systems can produce nominally deterministic instability through batch-dependent kernels, drifting logits, and deployment changes behind fixed model names.Batch-invariant kernels can restore bitwise equality at known cost, but fixed-input replays in this paper isolate serving instability rather than prompt variation.
  • Reproducibility: Existing reproducibility checklists require versions, hyperparameters, code, and compute but omit snapshot-identity levels, response-side model distributions, and measured metadata semantics.The paper identifies these omissions as load-bearing for API-hosted evaluation.
  • Positioning: The paper contributes no new judge or benchmark; it assembles a preregistered negative result, mechanism decomposition, prospective attribution, and reusable design rules.Its stated claim is the assembled whole, checked against its own archive at every number.

3 Measurement setup, preregistration discipline, and audit chain

The study measures a two-model pipeline in which a black-box observer evaluates visible prefixes of exact-arithmetic traces. Preregistered protocols, frozen gates, and append-only hashing define an external measurement instrument whose reliability is tested before scientific claims proceed.

  • 3.1 Measured object and operationalisation: Each task yields four explicit step-by-step candidate traces with exact rational answers, and correctness is assigned by canonical rational comparison.The only supervision label is exact final-answer correctness, not a model judgment.
  • 3.1 Measured object and operationalisation: The observer receives the task and a trace prefix truncated at 25%, 50%, 75%, or 100%, returning a correctness probability p_qnj.This operationalizes the measured system as an external visible-record pipeline.
  • 3.2 Three readout protocols (two instrument generations): The readout battery includes exact two-label protocols, dual-mapping adjacent pairs, and structured full rankings represented as exact permutations.The symmetrized protocol was introduced after a legacy label-semantics failure, while the judge protocol was selected on frozen development data.
  • 3.4 Preregistered gates: All design constants and instrument gates were frozen before paid calls, including a repeated-ranking Spearman median threshold of 0.90.The configuration covered task design, generators, observers, protocols, request plans, and budgets.
  • 3.4 Preregistered gates: The next-day replay gate required at least 24 hours and exact full-ranking permutation agreement of at least 0.99 across 100 byte-identical requests.All preregistered canaries remained in the denominator.
  • Audit chain: An append-only JSONL log records request and response hashes, timestamps, provider metadata, retries, and artifact manifests, with immutable completion records.Resumption is allowed only when the existing log is a strict prefix of the frozen plan.

4 Engineering correctness does not imply measurement reliability (C1)

Both preregistered campaigns achieved near-perfect execution integrity yet failed their frozen scientific reliability gates, separating engineering correctness from measurement reliability.

  • Clean failures, side by side: Both campaigns paired defect-free execution records with scientific gate failures, establishing the pattern as an instrument problem rather than a coding failure.The campaigns differed in instrument generation and failure axis, strengthening the evidential basis for the conclusion.
  • Frozen verdicts: 0.400 Spearman agreement failed the 0.90 I-3S threshold, while 0.78 replay agreement failed the 0.99 I-5R threshold.The same-window and next-day tests used different reliability axes, and the execution records remained at ceiling.
  • Preregistered discipline: The protocols retained drifted records, parse failures, and all valid groups rather than lowering thresholds, deleting cases, or switching metrics after observing results.The I-3S failure affected 29 of 31 valid rank groups, with every prefix level below 0.90.
  • Interpretation: Execution evidence such as delivery, schema validity, byte identity, and constant metadata is necessary for reproducibility claims but does not establish measurement-level reliability.The paper therefore requires reporting the engineering and measurement layers separately.

5 Three failure modes, one root-cause structure (C2)

The paper decomposes the shared-endpoint failure into three mechanisms: biased readouts, near-degenerate score gaps, and platform nondeterminism, which permutation rankings amplify into visible instability.

  • M1: biased instrument: M1 is a mapping-biased instrument: inverted label meanings produced a -0.632 median ranking correlation despite 0.0233 median absolute probability change.Responses emitted the requested labels correctly, so the defect was in the readout’s meaning assignment rather than token delivery.
  • M2: degenerate measurand: 0.400 repeat-ranking Spearman agreement coexisted with 0.00671 scalar repeat deltas because typical candidate gaps were far below the instrument’s noise floor.Repeat deltas were roughly 10^-3–10^-2 in probability, while the median minimum candidate gap was 3.04 × 10^-10.
  • M2: progress dependence: The ranking signal was concentrated mid-trace: Spearman medians were 0.000, 0.769, 0.500, and 0.000 at 25%, 50%, 75%, and 100% visible-prefix levels.At completion, readouts saturated and same-correctness candidates became indistinguishable again.
  • M3: unstable platform: Byte-identical next-day replays reproduced exact rankings 78/100 times, with drift across 16 of 25 tasks while provider-visible metadata remained constant.Same-day and cross-day agreement medians were 0.805 and 0.800, indicating that the 24-hour wait added nothing on the sampled platform class.
  • Root-cause structure: The tested gates were frozen before measuring either the gap distribution or platform noise, making their reachability an untested process assumption.The paper identifies the measurand, platform, and process as three distinct root causes.
  • Shared structure: The three defects converge because permutation readouts turn infinitesimal scalar perturbations across near-zero gaps into discrete ranking events.The paper’s controlled-error battery found separation by error type rather than error magnitude, limiting what the readout can measure.

6 The same drift under three metrics (C3)

The same replay drift looks radically different under full-ranking, top-1, and pairwise metrics; this spread is structural, not evidence that metric choice can repair the instability.

  • Three readings: The identical 100-replay batch yielded 0.780 full-ranking agreement, 0.950 top-1 agreement, and 0.985778 pairwise agreement.These are three aggregations of one experiment, not three independent experiments.
  • Why metrics diverge: Full-ranking agreement decays fastest because one adjacent transposition can invalidate the entire permutation even when most pairwise relations remain stable.The observed Kendall-distance spectrum was concentrated at small distances, consistent with frequent record-level violations amid strong pairwise stability.
  • Why metric choice matters: Preregistration preserved the full-ranking No-Go, while post-hoc metric selection would have supported a nearly stable narrative from the same replays.The protocols prohibited replacing the frozen verdict with top-1 or pairwise sensitivity readings.
  • Design rules: A reliability gate should match the granularity of the downstream claim, using equivalence bands rather than structurally fragile exact-match permutation gates where appropriate.A gate stricter than the claim can create unnecessary No-Gos, while a looser gate can support unsupported conclusions.
  • Design rules: Instrument redesigns can silently remove usable branches of a frozen gate, so the gate must be rechecked against the new readout before freezing.The earlier I-5 definition included a continuous probability-distance branch that the redesigned structured-judge readout no longer used.

7 More sampling cannot repair it (C4)

Increasing the number of binary observer samples reduced sampling error but did not make the frozen ranking gate reachable on the tested grid. The limitation arose because measured score separations were often negligible or exactly tied, rather than because sampling variance alone was too large.

  • Sampling does not repair the gate: The simulated audit used 500 replicates of the legacy D2 estimator with frozen D1-S readouts as explicitly declared probability proxies.The frozen within-task Spearman gate required a median of at least 0.90.
  • Sampling does not repair the gate: Between M = 8 and M = 500, sampling standard error improved sevenfold, but Spearman median rose only from 0.33 to 0.57 and the gate pass rate remained zero.Even the 95th percentile of the gate statistic was 0.80, below the frozen 0.90 threshold, at three-quarters of a million observer calls.
  • Why the pass rate stays zero: The median measured score separation was 3×10^-10 readout units, with exact ties in one-third of groups, so averaging could shrink noise but not create resolvable signal.For exactly tied groups, no estimator can order the pair better than a coin flip.
  • Boundary of the result: The conclusion is bounded to seven tested sampling scales through M = 500 and does not predict redesigned measurands with constructed, resolvable gaps.The passage characterizes this as an exploratory diagnostic rather than an executed experiment or authorization of D2.

8 What the platform’s own metadata can and cannot certify (C5)

Provider-exposed metadata did not certify replay stability: byte-identical requests could change rankings while visible fields stayed constant, were absent, churned, or failed to predict agreement. Consequently, reproducibility claims on shared endpoints are scoped to observed API behavior during the measurement window unless stronger snapshot identity is established.

  • What metadata cannot certify: 22 of 100 byte-identical requests returned different rankings even though response model, system_fingerprint, null-window fields, and finish reason were constant.A metadata-constancy check would nevertheless have certified the replay as clean.
  • What metadata cannot certify: The fingerprint field was absent for mistral and qwen, while openai produced 19 values in one day and 8 within a single 37-call window.Deepseek supplied one constant, information-rich value, showing that field behavior differed across providers.
  • What metadata cannot certify: Same-fingerprint pairs agreed no better than different-fingerprint pairs, at 0.854 versus 0.884, so the field did not predict stability.A constant fingerprint therefore did not distinguish stable from unstable replay behavior.
  • What metadata cannot certify: Under a constant fingerprint, full-agreement rate was 0.879, below the 0.99 replay requirement.A stable, information-rich deployment identifier still did not certify replay-exact behavior.
  • Snapshot-identity ladder: On shared endpoints, the reproducibility scope of a model name or model name plus fingerprint is the endpoint’s visible API behavior during the observation window.The paper therefore proposes reporting snapshot-identity level as mandatory metadata alongside temperature and seed.
  • Snapshot-identity ladder: L1 fixes provider-pinned weights but not necessarily observable behavior, whereas L2 adds self-hosted weights, batch-invariant kernels, pinned hardware, and framework versions.The L2 arm showed replay stability far above the shared-endpoint floor while the server was quiet, but exactness still required verification.
  • Snapshot-identity ladder: The self-hosted L2 test directly covered one model and one load pattern, with driver version unavailable, so its broader ladder recommendation remains bounded.The arm used recorded weights and a batch-invariant serving stack but was not pooled with shared-endpoint results.
  • Auditability boundaries: Capability probes found four documentation contradictions, one provider-level prohibition on temperature=0, incomplete rate-limit observability, and moving undated aliases.These boundaries make response-side model logging and measured parameter handling necessary for attribution.

9 Design rules for preregistered LLM evaluation (C6)

The paper converts the failed audit into design rules: establish the instrument’s noise and resolution before freezing gates, match metrics to claim granularity, aggregate repeated draws prospectively, and verify every load-bearing parameter. These rules target failures that occurred in the audited pipeline rather than offering abstract advice.

  • Identity and load: An L0 endpoint cannot support an I-5-class gate over a proposition no one controls; stability must be measured under representative load after snapshot selection.The recommended preference order is self-hosted open weights, provider-pinned snapshots, then shared endpoints with aggregated-readout gates.
  • Pilot before freezing: Both campaign-ending gates were frozen without piloting, and neither was reachable by the audited instrument class.The proposed repair is to pilot noise and gap distributions, simulate gate operating characteristics, and freeze only when discrimination is adequate.
  • Pilot before freezing: A ranking gate should include only pairs with pilot-measured gaps above a noise-derived resolution bound, excluding ties and near-degenerate pairs by preregistered rule.In this sample, 84% of groups were mostly below the detection limit, which should be reported as a sample property.
  • Match metric granularity: Exact-match gates amplify pairwise noise into record-level failure, as agreement changes from 0.986 pairwise to 0.780 at the record level.Continuous aggregates or explicitly aggregated per-candidate and pairwise scores reduce the impact of a single token flip.
  • Prospective aggregation: For full-ranking uses, freeze the number of draws, aggregator, tie rule, equivalence band, and TOST decision before execution rather than replacing the strict metric post hoc.The proposed approach keeps the full-ranking deliverable while making the aggregate the measured object.
  • Prospective aggregation: With Borda aggregation, exact agreement reached 0.94 at k = 3, 0.98 at k = 5, and 0.999 at k = 9 in the study’s calibrated simulation.Within-one-transposition agreement reached 0.999 by k = 5, making five to nine calls per measurement point sufficient in that simulation.
  • Prospective aggregation: On L0/L1 infrastructure, each call is one draw from platform nondeterminism, so every draw must be retained and aggregation rules must be fixed before execution.Replay-until-agreement, post-hoc majority votes, and dropping discordant draws condition on the verdict.
  • Construct resolvable tasks: Candidate sets should be manufactured to be resolvable through controlled errors, difficulty near accuracy 0.5, and mixed-correctness coverage above a preregistered floor.The study’s values, 0.781 and 0.163, should have disqualified the design during piloting.

10 Prospective supplements: decomposing, generalising, and calibrating the floor

Preregistered supplements decomposed the instability floor, tested whether it generalized across time and providers, and measured whether the judge could resolve known error gaps.

  • A-S1: same-day/cross-day variance decomposition: 0.805 versus 0.800 median full-agreement for same-day and cross-day pairs yielded d = 0.005, indicating no measurable waiting benefit in the sampled windows.The historical 0.78 lay within the same-day distribution, and the metric spectrum reproduced the same three observed magnitudes without pooling.
  • A-S1: same-day/cross-day variance decomposition: 0.8150 versus 0.8100 median agreement across five days replicated d = 0.0050, while per-day medians varied from 0.7800 to 0.8250.The replication rule fired even though one sampled day’s own floor exceeded the frozen 0.03 band.
  • A-S2: cross-provider stability: max = 0.879 < 0.95 across four providers fired the platform-class reading, with ordering stability ranging from 0.74 to 0.88.Every provider’s cross-day behaviour matched its same-day behaviour, and no provider approached replay-exactness.
  • A-S2: cross-provider stability: 0.775 versus 0.805 agreement for the qwen re-measurement and the primary A-S1 endpoint was read side by side, never pooled.The comparison concerns stability, not model capability or judging accuracy.
  • A-S3: controlled-error resolution: The controlled-error battery certified only that the detection limit lies below 10^-3 relative error.The battery did not bracket the detection limit, and extrapolation to the original candidates requires a separate argument.
  • A-S3: controlled-error resolution: Four of five slip types showed no separation increase with relative error ε, and one largest-error cell reached only 0.333, below chance.The pooled rise across bins was attributed mostly to slip-type composition rather than error magnitude.

11 Threats to validity and the boundary of claims

The paper bounds its claims to externally measured behaviour in specified configurations, with explicit limits on mechanism generalization, task transfer, and temporal interpretation.

  • External validity: Mechanism evidence M1 and M2 comes from one observer family, one shared endpoint, one synthetic task family, one prompt template, and one language.A-S2 broadens stability characterization to four providers, but does not extend the M1/M2 mechanism measurements.
  • External validity: The controlled-error detection limit applies to constructed slips with uniform wording on one provider in one session, not automatically to stylistically heterogeneous I0-R candidates.The four-pass replay disagreement is not a same-day stability measurement and is not compared with A-S1 or A-S2 values.
  • The measurand was ours: The near-degeneracy was manufactured by task design, with candidate accuracy 0.781 and mixed coverage 0.163, rather than established as a law of LLM-as-judge practice.The transferable hazard is failing to measure the measurand’s gap distribution before interpreting a ranking gate.
  • Scope boundary: The results address operational external measurements over visible records, not model internal states, representations, semantic dynamics, or named-provider service quality.Internal-theory questions were assigned to a separate white-box track.
  • Temporal scope: Same-day and cross-day claims sample only two days in r1 and five days in r2, so they do not extrapolate to any future day.The second round measured a sampled day whose own stability floor stood out.
  • Data and protocol limitations: A-S1 byte-equivalence with historical F2 wire bodies was asserted by construction rather than verified against archived bytes.This inherited limitation constrains the replay comparison’s evidential basis.

12 Disclosure, data, and reproducibility

The study preserves an auditable chain from preregistration through exclusions, archived evidence packages, reproducible figure rebuilds, and explicit limits on older runs.

  • Audit and disclosure: Every number resolves to a file in a frozen, hash-manifested archive, and every excluded run is disclosed with its reason and disposition.The governing index is itself version-pinned.
  • Audit and disclosure: Eleven exclusion events were retained and disclosed, while excluded runs remained outside every denominator and their costs stayed on the ledger.The exclusions ranged from duplicate candidates to aborted or voided supplement windows.
  • Audit scale: 52,988 attempts comprised 52,574 provider responses, 413 transport failures, and one residual event with neither response nor failure.These totals are reported to prevent audit volume from being mistaken for sample size or spend.
  • Reproducibility package: Every figure rebuilds from the public layer alone with one command and zero network calls.Restricted raw responses are integrity-linked through their SHA-256 hashes, while manifests, derived data, reports, configurations, and tests are public.
  • Reproducibility boundary: Nineteen phase-1 runs are capped at exploratory-diagnostic status because their exact executed configuration bytes cannot be reproduced.Only two phase-1 runs sealed inside their run directories with exact hash agreement qualify for stronger citation.
  • Cost disclosure: The entire F2 refutation window cost CNY 0.296479056, while the supplements added approximately 37.73.The paper reports these figures to distinguish evidence scale from spend and to document cost asymmetry.

13 Conclusion

The audits show that black-box LLM observers on shared endpoints failed preregistered instrument gates despite ceiling-level execution records. The evidence supports measuring snapshot identity, noise floors, score gaps, and metadata semantics before freezing evaluation criteria.

  • Two audits failed their instrument gates despite execution records at ceiling, with failures decomposing into biased readouts, near-degenerate separations, and replay instability.Metric substitution and scale did not repair the failures on the tested grid.
  • The proposed discipline combines snapshot locking, pre-freeze operating-characteristic calibration, detection-limit conditioning, continuous readouts, and prospective aggregation.The paper also recommends a reporting checklist and fail-closed guards for evaluation tooling.
  • The conclusions are scoped to black-box observation through shared serving infrastructure and do not assert claims about internal model dynamics.A separate white-box track is identified as having its own preregistration.
  • 0.005, from 0.805 − 0.800, was classified as a nondeterminism floor rather than a systematic cross-day component.The follow-up replicated this interpretation over additional sampled days.
  • 0.400 versus 0.90 was the I-3S repeat-ranking outcome, while 0.78 versus 0.99 was the I-5R next-day exact-replay outcome.The two gates assessed different instability axes: within-window ranking repeats and byte-identical replay across a 24-hour gate.

A.10 The excluded-data register

The excluded-data register records failed, voided, and permanently excluded execution events without repairing or silently merging them into study denominators. It also links these events to design rules intended to prevent recurrence.

  • Nothing was deleted, edited, or repaired; exclusions retained raw failure records and standing obligations.Excluded events were not used as capability evidence.
  • 40 of 80 tasks produced pairwise-duplicate candidates in Round 0, so all 160 generations were excluded and Round 1 reran with fresh identities and tasks.Nothing from Round 0 entered any denominator.
  • 100 A-S1 Day-1 calls yielded zero valid readouts because thinking mode remained enabled and responses hit the token limit.The window was disclosed as an instrument-failure exhibit and excluded from denominators.
  • The register converts observed failures into rules covering snapshot locking, pre-freeze calibration, detection limits, continuous readouts, aggregation, reporting, and fail-closed guards.These rules are tied to measured evidence and operational safeguards.
  • The reporting checklist distinguishes L0 model names, L1 pinned snapshots or dedicated deployments, and L2 self-hosted configurations with weights, framework, hardware, and determinism details.It also requires reporting response-side model strings, fingerprint status, parameter behaviour, and pilot-measured noise floors.
Loading 2609.04198v1…