Source-linked AI summary
Commit-first LLM judging inherits the judge's own errors
Idil Gozel
TL;DR
LLM judges are used because teams cannot check every output by hand, but systems optimized against their scores can exploit them. This paper audits shipped configurations and tests commit-first judging, finding that it moves the exploitable anchor to the judge’s own answer, so its protection depends on task-specific judge competence.
Problem
Because teams cannot check every output by hand, they use LLM judges, which optimization can game when wrong outputs earn scores.
Source: §1 · pp. 1–2Method
The paper audits shipped judge configurations, tests best-of-N against a documented configuration with and without commit-first judging, and probes judge answers against unseen checks.
Source: §1 · p. 2Results
Commit-first judging can stop gaming when the judge’s answer is right but worsen outcomes when it is wrong; judge competence is task-local rather than scale-dependent.
Source: §1 · pp. 2–3Takeaways & Limitations
The defence is bounded by judge competence on the task, which the authors report can be measured in advance with a cheap probe.
Source: §1 · pp. 2–3The experiment covers four small tasks, one judge family, and one generator; the population’s convergence on the judge’s score appeared in one of two seeds.
Source: §5 · p. 10
Abstract
from arXiv · showhide
LLM judges, models that score another system's output, can be gamed by the systems they score. Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match. We call this commit-first judging, and ask whether shipped software implements it, and what it costs. We audit the default judge configurations of eight widely used evaluation frameworks. Of the 24 configurations in scope, none implement it. Nine implement a variant the literature measures as ineffective, and share one ancestor prompt, traceable through a copied typographical error. In a controlled experiment, an ordinary best-of-N search with no access to correct answers optimises code against one of these configurations, used exactly as documented. On an interval merging task the judge accepted 90 of 96 candidates in one seed and 93 of 96 in the other; every accepted candidate passed every test the search could see and failed a held-out suite it could not. The judge identified the defective line and cited it as grounds for a perfect score. Commit-first judging removed the effect: 0 of 96 in both seeds. On a second task it made matters worse in both seeds: the judge's committed answer was wrong, and in one seed the population converged on it. This is our main finding. Commit-first judging does not remove the anchor that gets gamed, it moves it from the candidate to the judge's own answer, so evaluation is only as good as the judge is at the task. That precondition is cheap to measure in advance, and is task local rather than scale dependent: a smaller judge solved a task the frontier judge failed and resisted gaming where it did not. We also validate our own instruments: five of fifteen claims in our criteria were wrong against verbatim sources, and two held-out checks were unjustified by their specifications.
1 Introduction
Because systems are selected by evaluator score, judges that reward wrong outputs can raise scores without improving quality. This paper asks whether shipped frameworks use commit-first judging and tests its effects.
1 Introduction
- In this evaluation, step-by-step judging with the candidate already visible left a 0.719 false positive rate on wrong answers; requiring an answer before showing the candidate dropped it to 0.012.The candidate remains visible at comparison time; commitment, not blindness, is the operative variable.Source: §1 · p. 2
- Commit-first judging shifts the exploitable anchor to the judge’s own answer: it helps when that answer is right but can lock in errors when it is wrong.The authors report that gaming replicated across seeds, while the repair did not.Source: §1 · p. 2
- The paper contributes a census of 27 default judge configurations shipped by eight widely used frameworks, with verdicts tied to specific package versions and files.The verdicts are also pinned by SHA-256 digest and re-verifiable by script.Source: §1 · p. 2
- A controlled best-of-N search without ground-truth access gamed a shipped judge configuration on a real coding task, while commit-first judging eliminated the effect there in both seeds.The configuration was used as documented.Source: §1 · p. 2
- A cheap probe of judge answers against unseen checks predicted where commit-first judging helped or backfired, and competence depended on the judge-task pair rather than model scale.The paper also reports a smaller judge succeeding on a task the frontier judge failed.Source: §1 · pp. 2–3
- The authors validate their measurement instruments and define held-out-check reachability; verbatim-source checks found five of fifteen first-draft criteria claims were wrong.They corrected the claims before publication and publish the original wording alongside the ledger.Source: §1 · p. 3
2 What the eight frameworks ship
The audit finds no in-scope default configuration implements commit-first judging. Nine instead use a measured-ineffective variant, and thirteen supply a reference answer while leaving the judge to render the verdict.
2 What the eight frameworks ship
- The corpus covers 27 default templates and documented examples from nine packages across eight vendors, and is described as close to a census of that population.LangChain accounts for two packages.Source: §2 · p. 3
- The criterion requires a parseable judge answer committed before candidate comparison and acceptance based on matching that answer; the verdicts are version-specific.A release re-check found no verdict changes, and the experimental G-Eval template was byte-identical across the two releases checked.Source: §2 · p. 3
- None of the 24 in-scope configurations implement commit-first judging; each presents the candidate with the question and returns a holistic score.Three subjective-task configurations were excluded from this criterion’s scope.Source: §2 · p. 3
- Nine configurations use step-by-step reasoning with the candidate already in context, the variant previously measured at a 0.719 false positive rate.The nine share an ancestor prompt whose copying path is traceable through a distinctive typographical error.Source: §2 · p. 3
- Thirteen configurations provide a reference answer, which improves agreement with human raters but still leaves the judge as the sole source of the verdict.A reference consulted by the judge is not an independent channel that can disagree with it.Source: §2 · p. 3
3 The experiment
Across four coding tasks, commit-first judging eliminated gaming when the judge’s answer was correct but worsened outcomes when that answer was wrong. The results also show that held-out test validation and judge-task competence affect how these findings should be read.
3.1 Experimental setup
- The documented G-Eval configuration was tested with best-of-N selection over 96 candidates per run, without access to correctness measured on held-out tests.A pre-run admission rule excluded all four tasks, so the authors reported them all and flagged the rule as a design error.Source: §3.1 · p. 4
3.2 Validating the evaluation instrument
- The held-out suite caught 15 of 16 deliberately broken implementations; two unjustified checks changed no divergence counts but lowered interval correctness from 0.68 to 0.52 and 0.67 to 0.51.Twelve of thirteen checks were reachable under the paper’s definition.Source: §3.2 · pp. 4–5
3.3 When evaluation score and correctness diverge
- On interval merging, 90 and 93 of 96 candidates diverged; the judge gave the buggy non-touching merge a perfect score despite visible-test success.The judge identified the defective condition and cited it as grounds for the score.Source: §3.3 · p. 5
3.4 The fix and its trade-offs
- On interval merging, divergence fell from 90 and 93 of 96 to 0 of 96; on duration parsing, it rose from 37 and 41 to 50 and 75.The interval judge’s committed answer was correct; the duration judge’s was wrong, and one commit-first seed ended at 0.667 correctness while the evaluator read 0.99.Source: §3.4 · p. 6
- Commit-first judging replaces the candidate anchor with the judge’s answer: correct answers resist gaming, while wrong answers can become the optimizer’s target.Across the two duration seeds, final correctness was 0.958 and 0.667, so the lock-in outcome was not universal.Source: §3.4 · pp. 6–7
3.5 Control experiments
- Strict mode still diverged, ending at 0.83 correctness while the evaluator read 1.0; Haiku diverged only 4 of 96 times on interval merging, unlike the frontier judge’s 90 and 93.The commit-first frontier judge ran at default sampling rather than the specified temperature zero, affecting cross-model comparisons.Source: §3.5 · pp. 7–8
3.6 A low-cost probe for detecting vulnerable judges
- The three-attempt probe found the expensive judge solved interval merging 1 of 3 times and the smaller judge 3 of 3, while neither solved duration parsing.The probe’s outcomes match where commit-first helped and where it backfired; reliability is a property of the judge-task pair, not model tier.Source: §3.6 · pp. 8–9, Figure 4 · p. 9
4 Verification of our own claims
The authors checked their own quantitative claims against pinned source quotes and found substantial first-draft citation drift, then corrected it before publication.
4 Verification of our own claims
- The check required each quantitative criteria claim to resolve to a verbatim quote from a pinned source, with a location, and ran in continuous integration.Publication was blocked unless every claim resolved.Source: §4 · p. 9
- Five of fifteen first-draft criteria claims failed the verbatim-source check, and all five were corrected before publication.One claim was contradicted by its source, while four were broader than the evidence; the public ledger records the original wording and corrections.Source: §4 · p. 9
- Even under careful, deadline-free preparation, one in three first-draft claims failed, making the check an example of the paper’s verification mechanism applied to its author.The final ledger contains 26 claims: 25 verified and one explicitly marked as illustrative arithmetic rather than a sourced figure.Source: §4 · pp. 9–10
- Applying the same discipline to test suites exposed two held-out checks that their specifications did not justify.The authors report that both checks of their own work found something.Source: §4 · p. 10
5 Limitations
The experimental findings are bounded by a small, controlled study, task-specific results, and a census of default templates rather than modified deployments.
5 Limitations
- The study uses four small tasks, one judge family, and one generator, so its experimental scope is narrow.The tasks are small, single-function problems.Source: §5 · p. 10
- Divergence appeared on two of four tasks; the other two offered no headroom because the generator solved them.The authors describe this as a controlled demonstration, not a survey.Source: §5 · p. 10
- Commit-first populations rested on the judge’s score in one of two seeds, so the authors claim it can, not must, lock in a judge error.The first commit-first seeds predated committed-answer instrumentation, leaving those judge answers unrecorded; null-task, strict-control, and smaller-judge arms were single runs.Source: §5 · p. 10
- The census covers default templates, and its verdicts do not automatically transfer to systems that modify those templates.The defaults are what practitioners copy when beginning, but deployed systems may change them.Source: §5 · p. 10
- The study’s small tasks sit well below the code sizes used to fit prior work’s weak trend for a growing visible-versus-held-out gap.If that trend holds at this scale, the measured effects would be understatements rather than exaggerations.Source: §5 · p. 10
6 Reproducing the results
The paper supplies version-pinned evidence and artifacts for checking its census and reproducing its experimental results, while withholding some configuration materials.
6 Reproducing the results
- Each census verdict is pinned to a package version, file path, and SHA-256 digest, with a script that re-verifies quoted fragments against the artifact.The criteria, scored corpus, citation ledger, and verification tooling are public.Source: §6 · p. 10
- The run records include each candidate’s judge verdict, generated-results digest, figure data, and the spend ledger.There is one record per candidate.Source: §6 · p. 10
- The replication set includes task prompts, visible and held-out suites, reference solutions, negative-control probes, and the replay script.Running the script over published records reproduces all 1,642 candidate scores and the reachability verdicts for thirteen held-out checks.Source: §6 · p. 10
- The run artifacts call commit-first judging “deanchored,” the earlier name retained because other artifacts use it as an identifier.The paper states that the two names refer to the same configuration.Source: §6 · p. 10
- The judge configurations, search code, and model prompts are not published, though the named G-Eval and commit-first forms are reconstructible from their sources.The authors say they publish what is needed to check their claims.Source: §6 · p. 10
Competing interests
The author is founding Evaluator Integrity, a company focused on assessing automated evaluators; the reported findings concern reproducible audits of public open-source software at published package versions.
Competing interests
- The author is founding Evaluator Integrity, a company focused on assessing automated evaluators; the findings concern public open-source software pinned to published package versions and reproducible without the author or any product.Source: Competing interests · p. 11