PaperlayerView exact revision on arXiv

Source-linked AI summary

JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

Delip Rao, Chris Callison-Burch

arXiv:2609.29769v1cs.CL

TL;DR

The paper asks whether a typed classifier can replace an LLM rubric judge and examines how their judgments compare with human labels. It compares Jev with three LLM judges across nine benchmark panels, then studies cost, graded-scale disagreements, and cascades. Jev’s accuracy is often not distinguishable from an LLM judge’s, but correlated errors leave cascades with little accuracy to gain.

  • Problem

    Per-criterion LLM rubric judging requires one call per pair, motivating evidence on whether a typed classifier can replace an LLM judge and how judge shortfalls against human labels are shared.

    Source: §1 · p. 1

  • Method

    The study compares Jev with three LLM judges on nine benchmark panels using the same criterion texts, and replays Jev-first cascades on recorded verdicts.

    Source: §1 · p. 1, §6.3 · p. 13

  • Results

    Jev Choice ranks first among matched judges on RiceChem and second on HealthBench, while on graded panels it trails the best LLM judge; cross-fitted cascades gain at most 1.5 points over the best single judge.

    Source: §1 · p. 2, §8 · p. 15

  • Takeaways & Limitations

    For rubric-judge development, the results favor testing changes to judge instructions before adding judges given the same text, and reporting offsets and repeated errors alongside agreement with labels.

    Source: §8 · p. 16

    The study cannot distinguish omitted rater conventions, rubric wording, and shared judge training priors as explanations for graded-label differences.

    Source: Limitations · p. 16

Abstract

from arXiv · show

We ask whether Jev, a typed classifier that returns probabilities over permitted answers without generating text, can replace an LLM rubric judge. We compare it with three flash-tier LLM judges on nine panels drawn from seven benchmarks, giving every judge identical criterion texts. Jev's accuracy differs significantly from an LLM judge's in only 8 of 27 paired comparisons, ahead mostly on binary criteria and behind only on graded ones, and most of the other comparisons are inconclusive. Summed over the nine panels, the LLM judges, called once per criterion, cost 29 to 325 times as much as Jev and took 30 to 220 times as long. On graded criteria all four judges agree more with one another than with the labels and mostly assign lower levels than the raters. One of several observational accounts is that raters followed scale conventions our criterion texts omit. Jev's confidence ranks its own errors on most panels, which should make a cheap classifier the ideal first stage of a cascade that defers its uncertain verdicts to an LLM judge. Correlated errors undo that advantage. The LLM judges repeat nearly all of Jev's most confident errors, so a cascade replayed on the recorded verdicts lowers cost but gains at most 1.5 points over the best single judge with cross-fitted thresholds, and at most 2.0 even with oracle thresholds.

1 Introduction

The paper tests whether Jev, a typed classifier that returns probabilities over fixed answers, can replace an LLM rubric judge and whether their disagreements with human labels are shared. Across nine panels, Jev is often statistically indistinguishable from three LLM judges and substantially cheaper, but shared graded-scale errors limit the gains from cascading.

1 Introduction

  • The study compares Jev with three flash-tier LLM judges on 5,003 pairs from nine panels spanning seven benchmarks, using identical criterion texts.The account of graded-scale shortfalls is exploratory.Source: §1 · p. 1
  • Only 8 of 27 paired comparisons separate Jev from an LLM judge; its leads are mostly on binary panels and its deficits only on graded panels.Most remaining comparisons are too imprecise to establish equivalence.Source: §1 · pp. 1–2, Figure 1 · p. 2
  • Across graded panels, the four judges agree more with one another than with human labels, and on six of seven their mean levels are lower than the labels’.The shortfall is concentrated in a minority of criteria and mostly reflects scale placement; a post-hoc constant shift removes most of it.Source: §1 · p. 2
  • One observational account is that human labels reflect scale conventions absent from the criterion texts; shared training priors or rubric wording could also explain the pattern.No experiment varied the text, so these accounts are not distinguished.Source: §1 · p. 2
  • Jev’s confidence ranks its errors on most panels, but LLM judges repeat 96.0% of its most confident errors, and cross-fitted cascades gain at most 1.5 points over the best single judge.The cascade replays recorded verdicts and costs under half as much as the best LLM judge on most panels.Source: §1 · p. 2
  • The paper releases the judges’ verdicts and reports recommendations with evidence-status labels.Source: §1 · p. 2

2 Related Work

Prior work studies LLM judging, rubric design, and correlations with human ratings, but evaluation choices can obscure score differences or rater uncertainty. This paper positions its comparisons around exact agreement, offsets, and repeated errors as well as agreement measures.

2 Related Work

3 Judges, Panels, and Protocol

The study evaluates Jev and three LLM judges on nine benchmark panels against human labels, using paired comparisons and measures of agreement, offset, and cost. Its protocol specifies how each judge receives criteria and how labels and statistical comparisons are constructed.

3 Judges, Panels, and Protocol

  • Jev is tested as a replacement judge and as a confidence-based way to select pairs for LLM review.A Jev request carries all criteria for a unit, while an LLM judge is called once per pair; Jev returns probabilities or confidence without generating free text.Source: §3 · p. 3, §3.1 · p. 4

3.3 Panels

  • The evaluation uses nine panels from seven benchmarks, including two binary panels and seven mostly graded panels; the graded samples were fixed before judge calls.Judges read the study’s rubric versions, while original raters read benchmark instructions; no judge sees labels or sampling strata.Source: §3 · p. 3, §3.3 · pp. 4–5

3.1 Jev

  • Jev’s Noul primitive returns a yes probability, Choice selects the highest-probability supplied option, and Score returns an expected ordered level rounded to the closest level.Choice and Score use their own decoding rules; Jev Choice can include an NA option on graded panels, whereas Jev Score cannot abstain.Source: §3.1 · p. 4

3.2 The LLM judges

  • The three LLM judges run through the harness with default prompts, one call per pair, shuffled options, and provider-default reasoning settings.The harness prompts attention to negation, scope, and qualifying language and directs judges to choose the lower-quality level for an unresolved self-contradiction.Source: §3.2 · p. 4

3.4 Labels and reference points

  • Graded labels are rounded means of rater levels and binary labels are rater majorities or RiceChem TA annotations; the rater reference compares each rater with the other raters’ rounded mean.HelpSteer2 and LFQA also have no-read baselines: a most-frequent-level predictor and a provenance predictor, respectively.Source: §3.4 · p. 5

3.5 Metrics and statistics

  • The analysis reports exact and within-one accuracy, mean QWK, offsets, and accuracy gaps, and compares judges on jointly scored pairs using unit-bootstrap intervals to assess separation and parity.Separation requires a 95% interval excluding zero; parity requires a 90% accuracy-difference interval within 5 points of zero; showing neither is inconclusive.Source: §3.5 · p. 5

4 Jev’s Accuracy, Cost, and Speed

The section weighs Jev’s accuracy against its cost and speed, then examines alternative framings and abstention. Accuracy differences vary by criterion type and by which LLM judge Jev would replace.

4.1 Accuracy in paired comparisons

  • Of 27 Jev–LLM comparisons, eight separate after sampling-group resampling, five in Jev’s favor; most graded comparisons do not separate Jev Choice from an LLM judge.On the graded panels, 16 of 21 comparisons do not separate them, and most of those cannot establish parity either.Source: §4.1 · p. 6
  • On graded panels, Jev Choice is never the most accurate matched judge; Gemini is best or tied everywhere except LFQA, and on ELLIPSE even Gemini’s exact accuracy is 29.8% versus the 51.4% reference.On HelpSteer2 the 60.0% constant predictor beats every judge; on LFQA no judge differs significantly from the provenance baseline in exact accuracy.Source: §4.1 · p. 6

4.2 Jev is cheaper and faster than every LLM judge

  • Over nine panels, Jev Choice cost $0.063 and took 29 seconds; Luna, DeepSeek, and Gemini cost 29, 66, and 325 times as much and took 30 to 220 times as long.Gemini’s cost ratio generally rises with criteria per unit, but a one-call-per-unit harness or removing reasoning could narrow the ratios; neither alternative was costed.Source: §4.2 · pp. 6–7

4.3 Framing and abstention

  • Jev Noul and Jev Choice nearly agree on binary criteria, while Jev Score beats Jev Choice on within-one accuracy across all seven graded panels and is separated on four.On FED-Turn their exact-accuracy difference is 10.5 points, exceeding the 2.9-point spread among LLM judges; Jev Choice abstained on 28 of 3,414 ordinal pairs.Source: §4.3 · pp. 7–8

5 Shared Departures from the Human Labels

Across the seven graded panels, the judges share departures from the human labels: they agree more with one another, and their shortfalls largely reflect scale placement. Natural contrasts suggest omitted rating conventions as one possible account, but do not establish a cause.

5.1 Judge–judge agreement exceeds judge–label agreement

  • On all seven graded panels, judges agree more with one another than with the human labels; on ELLIPSE, all four are wrong on 64.0% of pairs.Even the least-agreeing judge pair exceeds the best judge’s agreement with labels on five panels.Source: §5.1 · p. 8

5.2 The gap concentrates in criteria with negative offsets

  • ρ = 0.92: across 31 graded criteria, mean judge offset tracks mean gap, and all four judges have negative offsets on 25 of 35 criteria.The correlation locates the gap but does not test its cause; Figure 2 shows offset against gap, with only FED-Turn engaging in the grey-guide region.Source: §5.2 · p. 8, Figure 2 · p. 8

5.3 A constant shift removes most of the gap

  • 13.4% to 50.4%: Jev Choice’s ELLIPSE exact accuracy rises after a post-hoc shift of every prediction up one level.After criterion-level shifts, 27 criteria are at most 5 points below the shifted rater reference; four remain more than 5 points below it.Source: §5.3 · pp. 8–9

5.4 Natural contrasts and candidate conventions

  • 59.4% vs 34.7%: PersonaChat raters assign top engaging levels more often than Topical-Chat raters despite identical criterion texts.A one-level-up shift raises every matched judge’s PersonaChat exact accuracy from at most 26.7% to at least 69.5%, partly mechanically because 63.3% of labels are at the top level.Source: §5.4 · pp. 9–10
  • On shared USR-TC responses, every judge over-grants MET on understandable and under-grants it on uses_knowledge, so error direction can follow the criterion.Within ELLIPSE, cohesion, whose descriptions name observable devices, is every judge’s best trait; every judge does worst on grammar’s quantifier-based levels.Source: §5.4 · pp. 10–11
  • ELLIPSE’s population-norm reading and LFQA’s top-level factuality contrast suggest missing conventions, but these natural contrasts do not establish causes.The ELLIPSE account may also reflect the “native-like facility” wording; LFQA raters grant the top factuality level more often than Jev Choice, while all three LLM judges object to false claims in the quoted answers.Source: §5.4 · pp. 9, 11

5.5 Label noise, the harness prompt, and shared priors

  • Label noise alone does not explain the errors on ELLIPSE, LFQA factuality, or USR-TC uses_knowledge; top-heavy scales can also yield negative offsets when judges use fewer levels.The latter pattern need not mean judges place units lower than raters.Source: §5.5 · p. 11
  • A shared reluctance to award high levels or overlapping training-data priors could explain common offsets; the evidence does not rule out either account.The harness prompt may add to LLM judges’ offsets, but it is not the common cause identified by the natural contrasts.Source: §5.5 · p. 11

6 Correlated Errors Undo the Cascade Advantage

Jev’s confidence ranks its errors on most graded panels, but LLM judges often repeat even its most confident mistakes. Replayed cascades save cost yet add little accuracy, and juries do not outperform the best matched judge.

6.3 Cascades lower cost but add little accuracy

  • Cross-fitted cascades improve on the best single judge by at most 1.5 points, while oracle-threshold gains reach at most 2.0 points and are not significant.The cascade falls below the best single judge on six of nine panels; the largest oracle gain is on HealthBench.Source: §6 · p. 11, §6.3 · p. 13

6 Correlated Errors Undo the Cascade Advantage

  • A cascade defers low-confidence pairs to a fallback LLM; its accuracy advantage depends on kept pairs where Jev is right and the fallback is wrong.Jev alone is right on only 12 to 37 pairs of a graded panel against the fallback, and often defers those pairs.Source: §6 · p. 11, §6.3 · p. 13

6.1 Jev’s confidence ranks its own errors

  • On six graded panels Jev’s confidence ranks its errors, with AUROC 0.57 to 0.70; ELLIPSE is the exception, with AUROC 0.49 and uniformly high error rates.Confidence also fails on individual criteria, including USR-PC engaging, where Jev is wrong on 83.1% of pairs at mean confidence 0.72.Source: §6.1 · p. 12

6.2 The LLM judges repeat Jev’s confident errors

  • In Jev’s top confidence band, LLM judges repeat its wrong answer on 80.5% to 92.8% of verdicts, versus 33.6% to 63.7% under the independence baseline.Across 84 confident graded-panel errors, they repeat Jev’s wrong answer 96.0% of the time, versus 50.3% under the baseline; many sampled pairs have split rater labels.Source: §6.2 · pp. 12–13

6.4 Juries do not beat the best single judge

  • The median of three LLM judges is never more accurate than the most accurate matched judge and trails by up to 12.8 points on ELLIPSE.Adding Jev as a fourth juror also never significantly improves on that judge; the median repeats Jev’s wrong answer on 82 of 84 confident-error pairs.Source: §6.4 · p. 14

7 Discussion

The discussion treats shared judge errors as a reason to compare judges with labels and with one another. It recommends clearer rubric instructions and cautions against attributing binary-panel patterns to scale type alone.

7.1 Judge validation and judge combination

  • The judges share offsets and confident errors; missing rater conventions, shared training priors, or rubric wording could explain the pattern, and the data do not distinguish these accounts.The paper recommends reporting repeated errors against an independence baseline when evaluating combined judges.Source: §7.1 · p. 14

7.2 Recommendations

  • The authors recommend testing exemplars or level descriptions naming observable features before adding judges given the same text; these rubric-writing recommendations were not tested by intervention.Evaluations should also separate unanimous from split pairs, report each judge’s framing and decoding rule, and compare judges on identical pairs.Source: §7.2 · p. 14, Table 5 · p. 15

7.3 Binary-panel expectations on graded panels

  • The binary-panel expectations may not extend to graded panels, but changes in domain, options, and instructions prevent attributing failures to scale type alone.Jev Choice abstained on 28 graded pairs despite never choosing CANNOT_ASSESS on the binary panels.Source: §7.3 · p. 15

8 Conclusion

Jev performs well relative to matched judges on binary checklist criteria, but trails the best judge on graded criteria. Cascades reduce cost but yield small accuracy gains, motivating improvements to rubric instructions and fuller reporting of judge errors.

8 Conclusion

  • On binary checklist criteria Jev Choice ranked first on RiceChem and second on HealthBench, while on graded criteria it trailed the best LLM judge on every panel.The graded-panel shortfall was significant on several panels, so Jev’s value there depends on which LLM judge it replaces.Source: §8 · p. 15
  • On graded panels, all matched judges mostly placed units lower than raters; missing conventions, shared training priors, or rubric wording could explain the pattern.The data cannot distinguish these accounts.Source: §8 · p. 15
  • Replayed cascades saved cost, improved on the best single judge by at most 1.5 points, and fell below it on six panels.The authors recommend testing rubric instructions such as exemplars or observable-feature descriptions before adding judges given the same text, and reporting repeated-error rates.Source: §8 · p. 16

Limitations

The findings have important limits from observational explanations, judge configuration, sampling, statistical choices, rubric-label alignment, and benchmark scope. These constraints limit how broadly the comparisons can be interpreted.

Limitations

  • The proposed scale-convention explanation is observational, and shared priors cannot be ruled out because vendors do not disclose training data.The conventions were inferred from errors; the suggested tests with exemplars, revised scales, or raters’ own instructions were not run.Source: Limitations · p. 16
  • The flash-tier, zero-shot setup does not measure the best obtainable LLM judge, and the study did not test a graded Jev wrapper or within-request criterion influence.Judges also received different instructions and option orders.Source: Limitations · p. 16
  • Jev and Luna were not deterministic, while Gemini and DeepSeek were not rerun; recorded verdicts therefore provide limited evidence about run-to-run variation.Across five RiceChem passes, about 3% of Jev’s pairs flipped at least once; a spot check found Luna agreed on 71 of 82 repeated pairs.Source: Limitations · p. 16
  • Per-criterion results rest on 25–258 units, and Jev Choice answered only 8 error_recovery pairs while the _nofact result used three contexts.The confident-error sample used 12 errors per graded panel, unlike Figure 4b’s analysis of every error.Source: Limitations · p. 16
  • Several analyses are post hoc, oracle-threshold cascade gains are optimistic, and cross-fitted estimates use halves as small as 12 units.The main intervals resample units rather than larger sampling groups; the 5-point equivalence margin is post hoc and lenient where accuracy is low.Source: Limitations · p. 16
  • Rubric versions depart from benchmark rubrics, some labels partly reflect other qualities, and tested graded panels are English with limited unit lengths.Nearly half of ELLIPSE labels depend on rounding; no judge was called by a dated model version, and prices reflect one date.Source: Limitations · pp. 16–17

Ethics Statement

The study uses public benchmarks without collecting new human-subject data and releases records so its reported numbers can be recomputed.

Ethics Statement

  • The study uses public benchmarks as released and collects no new human-subject data; HealthBench conversations serve only as evaluation material.The authors report no incentive from TypeSafe or its competitors.Source: Ethics Statement · p. 17
  • Judge costs are reported from recorded token usage and list or billed prices.Appendix B gives the cost accounting.Source: Ethics Statement · p. 17
  • Verdict records for every judge are released so the paper’s numbers can be recomputed without calling a model API.Appendix M lists the records and values requiring other inputs.Source: Ethics Statement · p. 17

Generative AI Use Disclosure

The appendix discloses the use of generative AI tools and documents panel construction, judging protocol, and cost accounting. It also reports panel-specific cost ratios and the handling of labels, failures, and repeated runs.

A Panels, Judges, and Protocol

  • The appendix specifies panel sampling, including proportional stratification for graded panels and whole-unit sampling for linked answers or responses.LFQA and USR panels used simple random samples; HealthBench stratified completions by theme and label composition.Source: A Panels, Judges, and Protocol · p. 19, Table A1 · p. 20
  • The harness structurally withholds labels and other identifying fields from judges; Jev receives context and submission but not criterion type.HealthBench’s physician-written ideal completion was removed from judge inputs except for an out-of-protocol ablation.Source: A Panels, Judges, and Protocol · pp. 19, 20, 22
  • The released labels and judge inputs follow explicit conventions, including dropping N/A ratings and using rater-majority labels for embedded binary criteria.The rater reference compares each binary rater with the majority of the others and skips ties; the table documents judge concurrency and cost scope.Source: A Panels, Judges, and Protocol · pp. 20–21, Table B1 · p. 23
  • DeepSeek’s calls were routed across 11 upstream providers and returned empty responses for six graded pairs, which were excluded from graded metrics.It also returned empty responses on three binary pairs; no Jev request failed.Source: A Panels, Judges, and Protocol · p. 22
  • Across five passes, 2.9% of RiceChem pairs changed verdict under Jev Noul and 3.3% under Jev Choice; every change lay within 0.1 of probability 0.5.Jev’s median request latency was 0.18 to 0.20 seconds per pass; Luna agreed on 71 of 82 repeated pairs.Source: A Panels, Judges, and Protocol · p. 22

B Cost and Wall Time

  • Jev Choice and Jev Score cost about $0.043 and $0.042 over the seven graded panels, while Gemini’s prompt-token price was about 18 times Jev’s.Gemini’s recorded cost is reproduced by $0.75 per million prompt tokens and $3.75 per million completion tokens.Source: B Cost and Wall Time · p. 22
  • DeepSeek’s billed cost was 1.34 times the harness estimate across seven graded panels; the billed totals exclude six failed graded calls and three failed binary calls.Panel factors ranged from 1.31 to 1.46.Source: B Cost and Wall Time · p. 22
  • 326× for Gemini on RiceChem, 396× on ELLIPSE, and 373× and 349× on the USR panels; unit length had little effect on Gemini’s cost ratio.RiceChem’s 6.8 criteria per unit also made it an exception to the criteria-count pattern.Source: B Cost and Wall Time · p. 22

C Paired Comparisons and Secondary Metrics

Paired comparisons yield selective, rather than uniform, differences among Jev and the LLM judges. Results vary with the comparison unit and metric, and resampling broader sampling groups widens some intervals.

Unit-level analysis

  • Bootstrap intervals and uncorrected pair-level tests each separated four comparisons in each direction, with seven of the eight separations coinciding.The exception was Gemini’s pair-level lead on FED-Turn; the intervals instead separated Luna’s lead on USR-PC.Source: C Paired Comparisons and Secondary… · p. 23
  • Unit sign tests were significant in six comparisons, three each direction; Holm correction retained the same three as the pair-level test.Two comparisons separated by intervals were not significant in the unit sign test.Source: C Paired Comparisons and Secondary… · p. 23

Single accuracies on ELLIPSE

Parity and equivalence

Resampling sampling groups

  • Resampling sampling groups widened nine of twelve intervals; resulting separations favoring an LLM judge were all Gemini’s.The percentile intervals are rough with only 12 to 45 groups.Source: C Paired Comparisons and Secondary… · p. 24

Jev Score against Jev Choice

Paired comparisons

  • After Holm correction, pair-level tests retained three separations: Gemini ahead on ELLIPSE, and Jev Choice ahead on HelpSteer2 versus Luna and RiceChem versus Gemini.The adjusted p-values for Jev Choice’s leads were 0.011 and 0.016, respectively.Source: C Paired Comparisons and Secondary… · p. 25, Table C2 · p. 25
  • 7 of 21 ordinal-pair comparisons found Jev Score significantly more accurate than an LLM judge, while 2 found it less accurate, both against Gemini.These 21 comparisons were not repeated at the unit level.Source: C Paired Comparisons and Secondary… · p. 26

D The Binary Panels

The binary-panel analyses show that a fixed wrapper can improve RiceChem accuracy but has mixed effects on HealthBench criteria. The section also examines Jev’s graded-score framing and abstentions, including how the NA option relates to raters’ responses.

D The Binary Panels

  • 9.6 points: on the full RiceChem set, the wrapper’s accuracy gain is 9.6 points, versus 10.7 on the panel.The wrapper lowers the false-positive rate from 61.1% to 34.9% while leaving the false-negative rate near 10%.Source: D The Binary Panels · p. 27
  • On HealthBench, the wrapper raises accuracy on 5 criteria, lowers it on 9, and leaves 20 unchanged.Each criterion contains only 4–25 pairs, so these differences show that a fixed wrapper can lower accuracy on individual criteria.Source: D The Binary Panels · p. 27
  • Jev Noul’s error rate falls from 46.1% on full RiceChem and 49.2% on HealthBench near a 0.5 margin to below 10% at margins of 0.4 or more.Figure D1a plots error rate by probability margin.Source: Figure D1 · p. 27, D The Binary Panels · p. 27

E Choice Against Score

  • On FED-Turn, Jev Score leads every matched judge in exact accuracy, within-one accuracy, mean QWK, and mean absolute error, but not offset.Gemini has the smallest offset; against leave-one-out targets, Jev Score’s exact accuracy is 66.3%, versus 63.8% for a crowd worker.Source: E Choice Against Score · pp. 28–29
  • On 215 ELLIPSE disagreements, Jev Score is higher on 179 and exact on 21.9%, versus 8.4% for Jev Choice; the higher framing gains when judges fall below labels.Every judge places ELLIPSE essays below their labels.Source: E Choice Against Score · pp. 29–30

F Abstention

  • Three criteria account for 27 of Jev Choice’s 28 abstentions: error_recovery, HelpSteer2 correctness, and LFQA factuality.Its mean NA probability is 0.57 on error_recovery, 0.11 on HelpSteer2 correctness, and 0.023 on LFQA factuality.Source: F Abstention · p. 30
  • Jev Choice abstained on all 7 dialogues with at least two N/A ratings, and its NA probability tracked workers’ N/A counts (Spearman ρ = 0.62).Raters gave 29 of 125 sampled ratings as N/A on error_recovery.Source: F Abstention · p. 30, Figure F1 · p. 31
  • The NA option may fit criteria that do not apply to a unit, but its contrast with CANNOT_ASSESS is confounded by scale, wording, rarity instructions, and panel.The study did not run both option wordings on both panel types.Source: F Abstention · p. 31

G The Offset in Detail

The graded-panel shortfall is largely associated with where judges place units on the scale, rather than how they order them. A post-hoc one-level shift removes much of the gap, but scale compression and several criteria remain important qualifications.

G The Offset in Detail

  • After the best shift, four criteria still stay more than 5 points below the reference when judges and raters are both shifted: informative, diverse, factuality, and engaging.The criteria are FED-Dialogue informative and diverse, LFQA factuality, and USR-PC engaging.Source: G The Offset in Detail · p. 31
  • Nearly every ELLIPSE error lies below the label: 99.5% for Jev Choice and 98.4% to 99.5% for the other judges, except DeepSeek at 96.9%.Jev Choice’s mean absolute error is 1.256 levels, while its mean signed error has magnitude 1.247 levels.Source: G The Offset in Detail · p. 31
  • Only 25.4% of individual ELLIPSE ratings are at levels 4–5, versus 36.3% of labels; no judge puts more than 9.3% of any trait’s predictions there.At level 1, raters used the level on 11 of 3,096 ratings, while judges predicted it on 30 (Gemini) to 376 (Luna) pairs.Source: G The Offset in Detail · pp. 31–32
  • The judges compress the scale: their pooled regression slopes range from 0.29 (Jev Score) to 0.44 (Gemini), versus 0.50 for one rater regressed on another.The rater slope is not a like-for-like benchmark because judges are compared with the less noisy two-rater label.Source: G The Offset in Detail · p. 32
  • Jev Choice’s offset grows from −1.02 levels in the lowest holistic band to −1.63 in the highest; this fits, but does not distinguish, a norm-based account.The same pattern for Gemini runs from −0.59 to −1.09 levels.Source: G The Offset in Detail · p. 33
  • +1 shift: every judge’s exact accuracy peaks at that shift, at 46.8% (Luna) to 59.6% (Gemini), while mean QWK remains below the rater reference.Mean QWK peaks at 0.31 to 0.48, versus 0.50 for the reference; Jev Choice rises from 0.15 to 0.34.Source: G The Offset in Detail · p. 33
  • On ELLIPSE, every judge is less accurate on half-way labels than other labels; Jev Choice scores 6.8% versus 19.3%.The half-up rule raises each trait’s labels by 0.19 to 0.27 levels against the unrounded two-rating mean.Source: G The Offset in Detail · pp. 31, 33

H Where the Conventions Live

The section examines candidate rubric conventions that may help explain judge–label mismatches and finds no consistent panel-level explanation from several measured properties. It also shows that confidence often ranks Jev’s errors, with notable exceptions at particular criteria and on ELLIPSE.

H Where the Conventions Live

  • An ELLIPSE grammar example fits a population-norm account: judges place an essay one level below raters, while the level descriptions give no base rate for errors among grade 8–12 learners.Rater training on that population would supply such a base rate.Source: H Where the Conventions Live · p. 34
  • On two LFQA factuality examples, all five judges choose “Partially accurate” where the label departs from the stated absolute top-level criterion.On HelpSteer2 complexity, judges make no level-4 or level-5 predictions in 450 predictions, though labels assign level 4 to 10 of 90 responses.Source: H Where the Conventions Live · p. 34
  • On USR-TC, _nofact contexts and understandable responses labelled UNMET account for 33 of Jev Choice’s 117 errors; 63 more are ordinal errors on split pairs.The remaining 21 errors are the only ones neither on a split pair nor under a candidate convention.Source: H Where the Conventions Live · p. 34
  • No-read baselines are competitive on HelpSteer2 correctness and LFQA: the provenance baseline reaches 68.1% accuracy, versus 61.1% for the modal-level predictor.HelpSteer2’s correctness text asks for fact checking, but its label tracks overall helpfulness (Pearson r = 0.94).Source: H Where the Conventions Live · p. 34
  • Measured panel properties do not consistently explain gaps: USR-TC and USR-PC share scales and criterion texts, yet Jev Choice’s accuracy gap is +0.3 points on one and −16.2 on the other.The two measures also place Jev Choice on different sides of the reference on the FED panels.Source: H Where the Conventions Live · pp. 34, 37
  • Criterion length shows no consistent relation with per-criterion accuracy; ELLIPSE is ambiguous because six traits share a template and differ in length by at most nine words.With six traits, length cannot be separated from what each trait’s level descriptions assess.Source: H Where the Conventions Live · pp. 34, 36
  • Unit length and rater expertise do not order the gaps consistently; across panels, the three shortest-unit panels range from 16.2 points below to 0.3 above the reference.Within panels, unit-length accuracy shifts lack a consistent direction.Source: H Where the Conventions Live · pp. 36–37

I Confidence

  • Outside ELLIPSE, Jev Choice’s error rate falls by 13.3 to 39.6 points from the 0.25–0.5 confidence band to the top band on every panel.On ELLIPSE, error rate is flat across bands; five criteria also have confidence AUROC below 0.45, including LFQA factuality and FED-Dialogue informative.Source: I Confidence · p. 37

J Repeated Errors and Juries

LLM judges repeat nearly all of Jev’s sampled confident errors, while median juries show little consistent improvement over the best matched judge. The repeated errors are concentrated in recognizable patterns across panels, and often occur where judges agree with one another but not with labels.

Repeated Errors

  • Luna, Gemini, and DeepSeek repeated 81, 80, and 81 of Jev’s 84 sampled confident errors, versus 126.7 repeats expected under independence.Jev Score gave Jev Choice’s answer on all 84 errors but is the same model asked a different question, so it is not counted as corroboration.Source: J Repeated Errors and Juries · p. 39
  • Across error types, judges often shared Jev’s wrong verdicts, including low essay levels, “No” dialogue predictions, and top-level coherence ratings.On ELLIPSE, Jev put all 12 erroneous essays at level 1; all five judges chose that level on 10, although nine labels were level 2 and three were level 3.Source: J Repeated Errors and Juries · p. 39
  • Outside ELLIPSE, all four matched judges were wrong on 17.2%–31.5% of ordinal pairs.This is the share of pairs on which choosing a correct judge whenever one exists would still leave no correct verdict.Source: J Repeated Errors and Juries · p. 39
  • On most panels, LLM error repetition rose with Jev’s confidence, though part of that trend also appeared in an independence baseline tied to criterion and label.Restricting counts to ordinal pairs or treating LLM abstentions and call failures as non-repeats left most confidence-band results similar.Source: J Repeated Errors and Juries · p. 39
  • On HealthBench’s 92 Jev Noul errors, the LLM judges repeated 55.8% of verdicts, versus a 27.5% baseline; for Jev Choice, the shares were 58.4% and 27.1%.Among 12 separately sampled HealthBench confident errors, the LLM judges repeated Jev’s answers in 32 of 36 verdicts.Source: J Repeated Errors and Juries · p. 39, Table J1 · p. 39

Juries

  • The upper four-judge median was most accurate on every panel except USR-TC, but its best result was only 1.0 point above the most accurate matched judge, with an interval of −2.3 to +4.3.The three-judge median exceeded the best judge on ordinal pairs only on USR-TC, by 1.4 points with an interval of −2.8 to +5.1; among 84 confident errors, the lower and upper four-judge medians were right on zero and one, respectively.Source: J Repeated Errors and Juries · p. 40, Table J2 · p. 40

K Cascades

Recorded-verdict cascades trade off the share deferred to an LLM judge against accuracy, but cross-fitted cascades using each panel’s best LLM fallback show little gain. Their gains depend on Jev’s advantage on kept pairs, while estimated costs assume deferred pairs have average length and reasoning-token use.

K Cascades

  • With each panel’s best LLM judge as fallback, cross-fitted cascade gains ranged from −2.1 to +1.3 points over that judge.Threshold and fallback choices were made on one half of each panel’s units and scored on the other; on six panels with oracle costs of 10%–59%, cascades matched the best LLM on only 46%–64% of held-out halves.Source: K Cascades · p. 40
  • A cascade’s gain over its fallback is bounded by Jev’s lead on kept pairs, because the fallback’s advantage on deferred pairs cancels out.On HelpSteer2, Gemini won deferred pairs 16 to 1, but Jev’s 54.5% on kept pairs was below Gemini’s 56.1%, leaving the cascade at 52.8% versus Gemini alone at 54.3%.Source: K Cascades · pp. 40, 42, Table K2 · p. 41
  • At the fixed graded-panel threshold τ = 0.75, cascade accuracy was below Gemini alone on ELLIPSE and FED-Dialogue; near-maximal curves were flat on several panels.With Gemini as fallback, accuracy stayed within 1.4 points of the oracle threshold from τ = 0.55 on FED-Turn, 0.30 on USR-PC, and 0.40 on HelpSteer2.Source: K Cascades · p. 42, Figure K1 · p. 42
  • With the best LLM fallback and oracle thresholds, reaching the fallback required deferring under 60% on HealthBench and most graded panels, but nearly every pair on ELLIPSE and FED-Dialogue.Where Jev alone matched or beat the fallback, including all three RiceChem fallbacks, the reach was zero; cost estimates assume each deferred pair costs the fallback’s average per pair.Source: Table K1 · p. 41, K Cascades · pp. 42–43
  • On LFQA, routing every factuality pair to Luna produced 69.8%, below the confidence cascade’s 72.1% and Luna alone’s 71.5%.Routing any one or two of the three criteria to Luna reached at most 70.9%.Source: K Cascades · p. 42

L Binary-Panel Expectations Re-examined

  • The binary-to-graded expectation table records whether each expectation holds, holds in narrower form, is reversed, fails, is undetermined, or was not examined.Expectations were written before graded-panel runs, but status assignments were made after analysis; binary and graded panels differ in several ways at once, so status does not identify which difference mattered.Source: L Binary-Panel Expectations Re-examined · p. 43, Table L1 · p. 44

M Release and Derived Statistics

  • The release includes judge verdicts and panel samples, and two scripts regenerate the paper’s numbers and figures from those files without calling models.The analysis script checks recomputed results against recorded results, including cascade curves and paired comparisons.Source: M Release and Derived Statistics · p. 43
Loading 2609.29769v1…