Source-linked AI summary
The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating
Gnaneswar Villuri, Hashmath Shaik, Alex Doboli
TL;DR
First-token logits offer a cheap verdict readout used by constrained decoding and likelihood-scoring harnesses, but the paper asks whether it measures position bias the same way as generating and parsing a verdict. Comparing both readouts shows that first-token figures overstate position bias when judges do not commit, motivating disclosure of compliance alongside those figures.
Problem
First-token logits are a cheap readout used in evaluation tooling, but their effect on measured position bias is an open question.
Source: §1 · p. 2Method
The study compares forced first-token logits with verdicts parsed after generation on the same response pairs, distinguishing pairs where the judge would lead with a verdict token.
Source: §2 · p. 2Results
Takeaways & Limitations
Report a judge’s compliance rate and identify which readout produced each position-bias figure; compliance costs one forward pass and no labels to measure.
Source: §7 · p. 4The cross-family comparison is a compliance contrast rather than an independent replication of the divergence, which was measurable only for Qwen3; the study tests open-weight judges at 8B and below.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Reading an LLM judge's verdict from the logits of its first generated token is cheap, requires no generation, and is exactly what constrained decoding and likelihood-scoring evaluation harnesses produce. We show that this readout distorts position bias in one direction: it overstates it in every condition we test, so figures obtained this way behave as upper bounds. The mechanism is that judges do not always lead with a verdict token, on 12% to 49% of pairs for three Qwen3 judges and under 3% for Llama-3.1-8B and Phi-3.5-mini, and forcing a read on those pairs returns whichever response was shown first rather than a judgment. Pooled over the 924 pairs where a judge did not commit, the forced read flips on 89.7% of them when the responses are swapped, against 47.5% read after generation (paired difference +0.422, 95% CI [+0.365, +0.467]). The distortion is specific to what is measured: it moves position bias by 42 points while moving judge accuracy by under one point in seven of ten conditions, so it misleads whoever audits a judge rather than whoever uses one. A second, smaller failure occurs even when the judge does lead with a verdict token, since it sometimes opens with one letter and reasons its way to the other, on 0 to 5.5% of pairs at a rate uncorrelated with compliance. We recommend reporting the rate at which a judge leads with a verdict token, which costs one forward pass and no labels, alongside any position-bias figure.
1 Two protocols, one name
Position bias is measured by how often a judge reverses its verdict when response order is swapped. Two common verdict readouts—generation and parsing versus first-token logits—are different operations, and the latter is used by evaluation tools.
1 Two protocols, one name
- Position bias counts how often verdicts reverse when response order is swapped.The paper describes this as presenting each response pair in both orders.Source: §1 · p. 1
- The generated read parses a verdict from text, while the cheaper first-token read selects between verdict-token logits without generating.Constrained decoding also yields a first-position verdict by masking illegal tokens.Source: §1 · p. 1
- First-token logits are used in production scoring and constrained decoding, prompting the paper to ask what that practice costs when measuring position bias.The paper notes it did not identify a published position-bias figure obtained this way.Source: §1 · pp. 1–2
2 Setup
The study compares forced first-token logits with verdicts parsed after generation on the same response pairs. It defines compliance by whether the judge would lead with a verdict token in both response orders.
2 Setup
- The forced read chooses the larger verdict-token logit at the first position, while the generated read parses explicit verdict patterns and retries once at 1,500 tokens if needed.A pair is compliant when the full-vocabulary argmax is a verdict token in both presentations.Source: §2 · p. 2
- The study tests five judges on JudgeBench’s 620 pairs and LLMBar’s 419 pairs, using X/Y, A/B, or 1/2 verdict tokens.Qwen3 thinking mode is disabled so the judges receive comparable single-turn prompts.Source: §2 · p. 2
- The design is conservative: a one-character prompt tests disobedience, and 88 presentations without a verdict after 1,500 tokens are excluded from both readouts.The second pass resolves 95% of Qwen3-8B cases but 54% of Qwen3-1.7B cases.Source: §2 · p. 2
3 Compliance varies widely and is not predictable
Compliance varies with the judge, verdict tokens, and benchmark, so it cannot be predicted from model identity alone. The Qwen3 results show substantial variation across token choices, benchmarks, and model sizes.
3 Compliance varies widely and is not predictable
- 0.506 to 0.676: changing only the two verdict tokens shifts Qwen3-8B compliance on the same benchmark.Compliance therefore depends on the model, prompt, and task together.Source: §3 · pp. 2–3
- 0.506 to 1.000: changing the benchmark while holding Qwen3-8B and its prompt fixed shifts compliance from JudgeBench to LLMBar.The passage describes LLMBar items as short instruction-following comparisons and JudgeBench items as technical questions inviting step-by-step work.Source: §3 · p. 3
- 0.663, 0.885, 0.506: Qwen3 compliance across 1.7B, 4B, and 8B is non-monotone, while Llama-3.1-8B validity was 99.8% under this prompt versus 9.5% under a reported multiple-choice prompt.A compliance rate cannot be inherited from another study.Source: §3 · p. 3
4 A forced read on an undecided judge locks to a slot
On non-compliant pairs, first-token logits produce substantially more swap-induced flips than generated verdicts because the forced read locks to a response slot. This one-sided distortion inflates position-bias measurements, while a separate, smaller mismatch can occur even when the first token is a verdict.
4 A forced read on an undecided judge locks to a slot
- 89.7% of 924 non-compliant pairs flipped under the forced read versus 47.5% under generation, a paired difference of +0.422 (95% CI [+0.365, +0.467]).Of the 420 pairs where readouts disagreed on a flip, 405 were flips seen only under the forced read and 15 only under generation.Source: §4 · p. 3
- On non-compliant pairs, the forced-read flip rate is itself the rate at which the readout locked to a slot, not evidence of a judge's slot lock.With two verdict tokens and no ties, a flip occurs exactly when the same slot is selected in both presentations; this identity held on all 5,720 pairs.Source: §4 · p. 3
- 827 of 829 locked non-compliant pairs selected the first position (0.998); in all five conditions, forced-read position-bias figures were upper bounds on generate-and-parse results.The lock was to a fixed position at the logit level rather than item-specific.Source: §4 · p. 3
- Accuracy was identical in four conditions and differed by more than a point in only three (largest difference: 6.7 points, 0.439 to 0.648 overall), while flip rate differed by up to 0.220.The added error depends on order, not the label, so the readout choice affects auditing more than use.Source: §4 · pp. 3–4
- On compliant pairs, first tokens disagreed with parsed final verdicts on 0–5.5% of pairs; across four judges, this rate was uncorrelated with compliance (r = 0.01).Phi-3.5-mini led with a verdict token on 97.1% of pairs yet changed its mind on 5.4%.Source: §4 · p. 4
5 A cheaper procedure, and what it is worth
A shortcut generates only when a judge does not lead with a verdict token, reducing generation substantially while retaining measured, imperfect fidelity. The resulting procedure is an approximation rather than a guarantee.
5 A cheaper procedure, and what it is worth
- 85.5% fewer order-presentations required generation: 1,678 of 11,596, by generating only when the judge did not lead with a verdict.Across ten conditions, the reduction ranged from 58.9% on the least compliant condition to 100% where the judge always led with a verdict.Source: §5 · p. 4
- Across ten conditions, substituted verdicts were wrong on 0–5.5% of compliant pairs, yielding fidelity from about 0.945 to 1.00.The shortcut escalates between two readouts of the same model and needs no calibration set.Source: §5 · p. 4
- The procedure is an approximation with a measured error rate rather than a guarantee.It reads the first token and generates only for orders where the judge did not lead with a verdict.Source: §5 · p. 4
6 Limitations
The divergence is measurable only in Qwen3, and the study covers open-weight judges at 8B and below. Its generated readout is also limited by a heuristic parser and token cap, while proposed compliance remedies remain untested.
6 Limitations
- The divergence is measurable only in Qwen3; the cross-family contrast reflects differing compliance, not an independent replication.The study tests only open-weight judges at 8B and below because frontier API judges do not expose logits.Source: §6 · p. 4
- The generated readout is a lower bound on eventual verdict recovery because it uses a heuristic parser and capped budget.Judge-position-bias reordering was not stable under bootstrap (50.1% of resamples), so the paper makes no reordering claim.Source: §6 · p. 4
- The question concerns a conditional practice, not correction of a published number; output prefilling is a plausible but untested remedy.Prompts engineered for single-token output may reach compliance close to 1, but neither this nor prefilling is usually verified or reported.Source: §6 · p. 4
7 Recommendations and conclusion
The paper recommends disclosing compliance and readout choice when reporting judge position bias. First-token verdicts can be unreliable when judges do not lead with a verdict token, and both relevant rates can be measured in advance.
7 Recommendations and conclusion
- Report compliance rate: it costs one forward pass per pair, needs no labels, and varies from 0.506 to 1.000 across conditions.The paper also recommends reporting which readout produced the verdict and treating answer format as an experimental variable.Source: §7 · p. 4
- A constrained-decoding figure and a generate-and-parse figure measure different quantities when compliance is incomplete.The paper says first-token reads are convenient but unreliable.Source: §7 · p. 4
- Up to 49% of pairs lack a first-token verdict, and up to 5.5% of apparently compliant first tokens are not the judge’s conclusion.Both rates are measurable in advance, yet neither is reported.Source: §7 · p. 4