JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Introduction method

• JEV returns a typed verdict and probabilities for each label without generated text; its largest label probability q determines whether to accept the verdict or escalate it to a stronger judge.The paper evaluates accuracy, probability quality, fees, and latency, noting that typed probabilities alone do not guarantee accuracy or calibration.Source: §1 · p. 2, Figure 2 · p. 2

Introduction study design

• The comparison covers sixteen other judges across pairwise preference, evidence-grounded factuality, and final-answer adjudication, with blinded human audits of disputed labels in a 990-item subset.The study also includes follow-ups on four-way selection, answer style, answer format, and natural prose, plus a pre-specified live test on two new workloads.Source: §1 · p. 2

Introduction results

• Across 4,850 public judgments, JEV ranked eighth of 15 LLM judges, 1.7 points behind GPT-6, while costing 277 times less and answering in 0.15 seconds at the median.JEV was within three points of GPT-6 on ordinary preference and final-answer adjudication, but trailed by 7–28 points on expert knowledge, code, math, and logic puzzles.Source: §1 · p. 2
• At confidence of at least 0.9, JEV and GPT-6 were nearly tied (94.7% versus 94.4% on 3,744 items), but confidence weakened on style-adversarial pairs and reference-free prose.Confidence AUROC for detecting JEV errors was 0.76 on style-adversarial pairs versus 0.84 on style-matched pairs; every tested LLM judge was near chance yet confident on reference-free prose.Source: §1 · pp. 2–3
• A frozen cascade beat GPT-6 by 0.9 points on 1,610 held-out pairs at 41% of its fee.In a pre-specified live test on two new workloads, it matched GPT-6’s accuracy exactly while escalating 74% of pairs and saving a quarter of its fee.Source: §1 · p. 3

LLM judges and presentation bias

• Prior studies document position, verbosity, and self-preference biases, while mitigation configurations do not suit every judge; this work tests sensitivity under one verdict contract.The tested sensitivities are order, paraphrase, and style.Source: §2 · p. 3

Efficient and specialized judges

• Rather than add another specialized evaluator, this study profiles a hosted decision-only judge across workloads, measuring accuracy, probability quality, fees, latency, and escalation potential.The related work includes reference-based classifiers, small fact-checkers, single-token evaluators, and typed-decision checkpoints.Source: §2 · p. 3

Reward models and benchmarks

• The evaluation includes PairRM and Skywork-Reward-V2, covers five judge benchmarks, and uses PPE for the prospective test.The five benchmarks are RewardBench, JudgeBench, HaluEval, RewardBench 2, and RM-Bench.Source: §2 · p. 3

Confidence and escalation

• Prior confidence-based cascades motivate selective escalation, but a JEV-first replay barely beats the best single judge; this study tests escalation to a stronger reasoning judge.Prior work includes calibration and selective classification, cascades for reducing generation costs, and routing uncertain comparisons to stronger judges; the paper also tests its cascade live on new workloads.Source: §2 · pp. 3–4

Tasks and protocol

• Judges evaluate pairwise preference, evidence-grounded factuality, and final-answer adjudication, returning a verdict and probability for every allowed label without rationale.The verdict is the most probable label, and its probability q is used as confidence; tasks range from text-readable verdicts to derived judgments such as math correctness.Source: §3 · p. 4

Public benchmarks

• Public benchmarks include 1,500 RewardBench pairs and 350 JudgeBench pairs for preference, plus 3,000 HaluEval judgments for evidence-grounded factuality (Li et al., 2023).RewardBench sampling does not reproduce official subset weighting and its labels have mixed origins; the complete GPT-4o JudgeBench split covers knowledge, reasoning, mathematics, and coding (Tan et al., 2025).Source: §3 · p. 4

Final answers and controls

• Final-answer adjudication uses 150 replies with correct, incorrect, or no-answer labels, but undocumented annotators mean results measure agreement rather than verified accuracy.Controls comprise 108 reference-correct GSM8K replies and 64 synthetic evidence judgments spanning support, contradiction, missing information, and distractor cases.Source: §3 · p. 4

Splits

• The 4,530-item held-out set shares no source questions with the 642-item pilot; its 1,610 preference pairs test routing policies without entering a fit.The pilot was frozen before inference; its public tasks were split by source question into a selection set and a pilot test set.Source: §3 · p. 4

Collection rounds

• The study collected judges in successive rounds, while frozen routing policies, human-adjudication outcome, and prospective test were specified before evaluation data or model calls.The prospective round tested the recommended procedure on two new workloads; multi-family comparisons, post hoc threshold sweeps, and follow-ups were exploratory.Source: §3 · pp. 4–5

Diagnostics

• Diagnostics reverse all 1,850 preference pairs and repeat or paraphrase requests on 48 fixed examples; PairRM also judges both pair orders.Each example set spans 16 items per public task, with two repeated requests and one paraphrased-rubric request.Source: §3 · p. 5

Judges and measurement

• JEV confidence is q = maxk pk, the probability of its verdict, rather than its JEV-only native confidence field.JEV’s Choice, Noul, and Score types return probabilities over labels, yes/no, and ordered rubric levels, respectively; experiments generally use one Choice question per request.Source: §4 · p. 5
• Comparisons cover seventeen configurations, including thirteen hosted judges and four local baselines, under a shared output contract for generative judges.The local Qwen3 and Qwen3.5 checkpoints have no API-equivalent price; reward-model comparisons use distinct scoring procedures, and Laya checkpoints appear only in the appendix.Source: §4 · p. 5
• Invalid outputs count as accuracy errors, while probability metrics use valid outputs; paired differences use 2,000 source-question-clustered bootstrap resamples.Reported metrics include accuracy, macro-F1, Brier score, clipped NLL, ECE, and error-detection AUROC.Source: §4 · p. 5
• Latency is measured on a frozen 120-judgment panel, while fees use reported usage at collection-time prices and conservatively reserve for missing usage.Outcome latency includes network, provider, and retry time, not intrinsic inference time or throughput; dollar values are estimates, not invoices.Source: §4 · pp. 5–6

5 Overall performance, cost, and latency

• JEV ranks eighth of fifteen LLM judges on 4,850 pooled public judgments, 1.7 points behind GPT-6 and 2.8 behind GPT-5.6.GPT-5.6 Sol is 1.1 points higher than GPT-6 on the pooled judgments at half the fee; GPT-5.2 and GPT-5.4 are also slightly higher. GPT-6 leads on final answers and ties for best on JudgeBench, while Gemini 3.1 Pro and GPT-5.6 lead RewardBench and HaluEval, respectively.Source: §5 · p. 6
• JEV costs $0.044 per 1,000 judgments and has 0.15 seconds median latency, about 277 times cheaper and 13 times faster than GPT-6.JEV’s p95 latency is 0.24 seconds, compared with 3.86 seconds for GPT-6 and up to 23.2 seconds for Qwen3.6. Among hosted judges under $1 per 1,000 judgments, JEV is most accurate on all three public tasks, by 0.1 points over GPT-OSS on HaluEval and 0.4 points over Gemini 3 Flash on RewardBench.Source: §5 · p. 6
• JEV is mid-pack in accuracy but at the far end in cost and speed.Its accuracy gap is concentrated in particular skills and at middle difficulty.Source: §5 · p. 6

Ordinary preference, factuality, and final answers

• 92.5% each on 1,500 RewardBench pairs; JEV scores 87.3% versus GPT-6’s 88.4% on HaluEval and 94.0% versus 96.7% on final answers.Human adjudication favors GPT-6 by 3.0 points on RewardBench and 2.5 on HaluEval; 24 of 26 jointly missed HaluEval items had unsupported labels.Source: §6 · p. 6

Difficult correctness

• 78.6% versus 93.1% for JEV and both GPT-6 and GPT-5.6 on JudgeBench, a paired difference of −14.6 points.Human adjudication widens the difference to −16.0 points; reasoning, coding, and math disagreements go 27–0, 9–0, and 7–0 for GPT-6, and JEV still trails on similarly sized answers.Source: §6 · pp. 6–7

Reading versus deriving

• Within about two points either way, JEV matches GPT-6 on chat quality, harmful-request refusals, requests that should be answered, and evidence-grounded factuality, but trails on derived results.Pooled gaps are −7.0 points on expert knowledge, −12.9 on code, −14.3 on math, and −27.6 on logic puzzles; the gap is concentrated at middle difficulty, where JEV trails by 13.1 points when 3–5 other judges err and 15.3 when 6–9 err.Source: §6 · pp. 7–8, Figure 4 · p. 8

Misleading style and harder selection

• −8.8 points: JEV falls from 85.4% to 76.6% when the rejected answer is more elaborately written, while GPT-6 falls 1.7 points.On four-way RewardBench 2 selection, JEV scores 73.0%, GPT-6 75.0%, and Skywork 79.0%; style-adversarial pairs therefore expose a different weakness from answer selection.Source: §6 · pp. 8–9, Figure 4 · p. 8

Which way JEV errs

• On HaluEval, JEV catches hallucinations at 80.1% versus GPT-6’s 79.6%, but accepts fewer correct answers: 94.4% versus 97.1%.Length is not generally the driver: within-benchmark gaps barely change across input lengths on RewardBench and HaluEval, while they widen on long JudgeBench and RM-Bench items; JEV’s position sensitivity also varies by benchmark.Source: §6 · p. 9

Answer format

• −5.3 points: on 150 final-answer replies, JEV agreement falls from 91.3% under direct adjudication to 86.0% under extraction, while GPT-4.1 mini remains at 87.3%.On 40 HaluEval questions, rendering replies as multiple choice rather than free response lowers JEV agreement from 100.0% to 92.5%. JEV satisfies the output contract on every base item; failed calls count as errors, although they can reflect service failures rather than wrong judgments.Source: §6 · pp. 9–10

Natural prose

• 69.8%: JEV matches labels on document-grounded summaries, versus 53.5% on reference-free prose, where all three tested judges are near chance yet confident.JEV’s reference-free prose Brier score is 0.821 and error-detection AUROC is 0.498; the samples differ in content and label provenance, so they show workload dependence rather than a pure evidence effect.Source: §6 · p. 9

The operating envelope

• Within three points of the comparator on ordinary preference, evidence-grounded QA, final answers, four-way selection, and grounded summaries, JEV trails by 6–15 points on difficult correctness and RM-Bench, and is near chance on reference-free prose.The operating map recommends inexpensive judging for the first group, local validation for the second, escalation to a stronger judge for the third, and no tested judge for reference-free prose.Source: §6 · p. 10

7.1 Confidence locates JEV’s errors

• At q ≥ 0.9 JEV’s single-call error rates were 2.1% on RewardBench and 6.5% on JudgeBench, versus GPT-6’s 3.5% and 2.0%.Confident errors were 21% and 12% of JEV’s errors on those tasks, versus 41% and 25% for GPT-6. HaluEval confident-error counts are harder to interpret because many supplied-label errors are label noise.Source: §7.1 · p. 10
• At q < 0.6 JEV is correct on 55.1% of pooled items, versus 97.8% at q = 1; task AUROCs are 0.88, 0.73, and 0.83.At τ = 0.9, the judges were nearly tied on accepted items (94.7% versus 94.4%), while GPT-6 led on escalated items (75.0% versus 67.7%).Source: §7.1 · p. 10, Figure 5 · p. 11

7.2 Probability quality

• JEV’s single-call error-detection AUROC was 0.875, 0.745, and 0.827 on RewardBench, JudgeBench, and HaluEval, respectively, versus GPT-6’s 0.894, 0.907, and 0.831.Brier scores were 0.113/0.297/0.196 for JEV and 0.115/0.095/0.213 for GPT-6 across those tasks; on adjudicated HaluEval answers, they were 0.062 and 0.032. Pilot-fitted temperature scaling improved held-out HaluEval NLL from 0.496 to 0.304 but worsened JudgeBench (0.449 to 0.475) and RewardBench (0.192 to 0.216), showing that fits need held-out validation.Source: §7.2 · p. 11

7.3 Is the signal stable?

• JEV changed no decisions across 96 repeated requests but changed 4 of 48 paraphrases; its Choice, Noul, and Score probabilities only approximately agreed.Choice and Noul probabilities differed by 0.055 on average, Choice and Score by 0.015, and the two Noul probabilities missed summing to one by 0.045; thresholds should be tuned for one fixed output type. Between collection rounds, 7 of 350 JudgeBench decisions changed, moving accuracy by one item.Source: §7.3 · pp. 11–12
• At τ = 0.9, 92 of JEV’s 95 reversal flips fell below the gate; only 3 of 1,293 pairs at q ≥ 0.9 flipped.Averaging both orders raised pairwise accuracy from 89.89% to 90.65% and lowered Brier from 0.148 to 0.141, but AUROC moved from 0.863 to 0.858. In the pooled cascade sweep, both-order routing retained 100.2% of GPT-6’s accuracy at 33% of its fee, versus 100.2% at 31% for base-order routing.Source: Figure 7 · p. 13, §7.3 · p. 13

7.4 Where the signal weakens

• Confidence routing weakens on hard style-adversarial RM-Bench pairs and fails on reference-free prose, where AUROC is 0.498.On hard pairs, a τ = 0.9 cascade retained 98.9% of GPT-6’s accuracy (89.1% versus 90.1%; −1.00 points), while τ = 0.99 matched GPT-6 at 90.1% but escalated 76%.Source: §7.4 · p. 14

8 Accept when confident, escalate when unsure

• The cascade’s value depends on complementary errors: GPT-6 corrected 60 of JEV’s 75 JudgeBench errors, while 15 errors were shared and therefore unrecoverable by escalation.Unlike fusion, which calls every judge on every item, the cascade pays for the stronger judge only on escalated items.Source: §8 · p. 14
• At τ = 0.9, the post hoc pooled cascade scored 90.2% versus GPT-6’s 90.0% at 38% of its fee; random escalation at the same spend reached only 89.0%.The cascade escalated 23% of items; its paired difference was +0.21 points, with interval [−0.10, 0.49].Source: §8 · pp. 14–15, Figure 8 · p. 15
• On other workloads, routing matched or exceeded GPT-6 selectively, including 98.3% of its accuracy on HaluEval at 20% of its fee and 75% accuracy on four-way RewardBench 2 at τ = 0.5.On 240 human-corrected HaluEval answers, τ = 0.8 matched GPT-6’s 98.3% at 13% of its fee; on RewardBench, the cascade scored 93.8% versus 92.5%.Source: §8 · p. 15
• A cheaper fallback can improve the tradeoff: JEV→GPT-5.6 reached 91.2%, above GPT-6’s 90.0% and GPT-5.6’s 91.1%, at 17% of GPT-6’s fee.Alternative first stages showed benchmark-dependent outcomes, and the retrospective comparison does not establish a global cost–accuracy frontier.Source: §8 · pp. 15–17
• 93.4% versus GPT-6’s 92.5%: the frozen JEV→GPT-6 policy gains 0.93 points on 1,610 held-out pairs at 41.4% of GPT-6’s fee.It accepted 68.5% of pairs; the escalated 31.5% contained 85% of JEV’s errors.Source: §8 · p. 15, Table 3 · p. 16
• On RewardBench, the cascade beats GPT-6 by 1.27 points at 26.7% of its fee, but on JudgeBench it remains 0.74 points below GPT-6 at 66.8% of its fee.On JudgeBench, escalation closes 94% of the JEV-to-GPT-6 accuracy gap; for GPT-5.6, the selected policy instead loses 4.81 points there, illustrating that the selection rule does not always transfer.Source: Table 4 · p. 16, §8 · p. 16
• In the live replication, the cascade scored 92.5% against GPT-6’s 93.1% at 57.2% of its fee, with 0.27-second median latency versus 2.10 seconds.It escalated 46.5% of pairs; the replication confirms the offline simulation, not the policy’s generality.Source: §8 · pp. 17–18

How many local labels does a threshold need?

• Threshold choice must use target-task labels: the lower-bound rule reduces losses exceeding two points to about 5% with 96 or 192 labels, and separate benchmark thresholds reduce them further.The rule selects the most permissive threshold whose accuracy loss stays within two points; with 96 labels, separate RewardBench and JudgeBench thresholds reduce the rate to 0.6%, at median fee ratio 0.34. The simulation reuses items, so may understate distribution-shift risk.Source: §8 · p. 14, How many local labels does a threshold… · p. 18

9 A prospective live test

• The frozen protocol tested 500 PPE correctness pairs and all 270 pairs in JudgeBench’s Claude-3.5-Sonnet split, selecting thresholds on seeded sets of 100 pairs.For each workload, the lower-bound rule selected from thresholds 0.5 through 0.99; the cascade called GPT-6 when JEV was unsure and GPT-6 also judged accepted pairs for scoring.Source: §9 · p. 18
• 88.2% on PPE and 94.7% on JudgeBench: the live cascade matched GPT-6’s accuracy exactly on both held-out workloads.It escalated 68% of PPE pairs and 89% of JudgeBench pairs, caught 93% of JEV’s errors, and cost 75.5% of GPT-6’s fee ($12.5 versus $16.6 per 1,000 pairs). Median latency was 1.87 seconds for the cascade versus 1.91 seconds for GPT-6 alone. Parity also held on JudgeBench pairs with unseen and seen questions.Source: §9 · p. 18
• At τ = 0.9, the frozen general threshold escalated 60% of pairs, cost 62% of GPT-6’s fee, and lost 0.5 points overall and 1.2 on JudgeBench.At τ = 0.8, it lost 3.0 points at 44% of GPT-6’s fee; below that, JEV’s weakness showed through. Reading JEV’s base order alone halved calls and nearly traced the same curve, matching GPT-6 at τ = 0.99 (+0.2 points) at 84% of its fee.Source: §9 · p. 18
• JEV’s confidence gate was weakest on IFEval formatting-constraint pairs, with AUROC 0.57 versus 0.69–0.81 on the other PPE sources.The live run produced no invalid output and cost $12.77.Source: §9 · p. 18

10 Conclusion

• At 41% of GPT-6’s fee, the confidence cascade was 0.9 points more accurate on 1,610 held-out pairs and matched GPT-6 exactly on two new workloads.On the new workloads, JEV alone trailed by 14 points.Source: §10 · p. 19
• At 0.36% of GPT-6’s fee and 0.15-second median latency, JEV came within three points of GPT-6 where verdicts were readable from text.This included ordinary preference, safety, and evidence-grounded factuality.Source: §10 · p. 19
• JEV trailed GPT-6 by 7–28 points where verdicts had to be derived, while confidence routing weakened on style-adversarial pairs and failed on reference-free prose.A more elaborate wrong answer could mislead JEV, though its misses fell mostly among low-confidence decisions.Source: §10 · p. 19
• For local validation, average verdicts over both presentation orders, set the threshold using about 100 local labels and the lower confidence bound of accuracy loss, then check held-out items.When the first stage is weak, this procedure spends more rather than losing accuracy.Source: §10 · p. 19

Limitations

• Evaluation covers one proprietary JEV and selected non-compute-matched judges, with unknown benchmark training overlap and no tests in specialized professional domains.Decision-only JEV omits explanations and extended deliberation; Laya results cover two zero-shot checkpoints without task-specific fine-tuning. Escalation is serial to an independent judge, and thresholds target fixed accuracy tolerance rather than the costs of wrong verdicts and escalation.Source: Limitations · p. 19
• Only frozen two-order policies, the primary human-adjudication outcome, and the prospective test were prespecified; slice analyses, threshold sweeps, and the alternative-first-stage comparison were retrospective.Held-out pairs share benchmark families with fitting data, only 270 are from JudgeBench, and intervals are not adjusted for multiple comparisons.Source: Limitations · p. 19
• Primary results use supplied benchmark labels; human adjudication covers 183 items from a 990-item subset selected by judge disagreement, annotated by one team member, and probability-score conclusions vary with label quality.Source: Limitations · p. 19
• Latency reflects one client location and collection period, fees follow documented pricing rules rather than invoices, and most cascade cost and latency estimates are simulated offline.Exceptions are the live replication and deployed thresholds of the prospective test, which covers two correctness workloads and 570 held-out pairs.Source: Limitations · p. 19

Ethical considerations

• The study reused benchmark responses and annotations, adding 183 blinded team adjudications; it cautions against using automated judges as the sole arbiter of consequential decisions.Public redistribution requires provenance and licensing review; API credentials and account metadata are excluded from the manuscript and supplement.Source: Ethical considerations · p. 19

A Data, collection rounds, rubrics, and request examples

• The routing rule and threshold grid were fixed before the main round and held-out outcomes were not used for fitting.The held-out set shared no source question with the pilot, and repeated answers to one source question stayed together. Enlarged samples were collected under a frozen protocol with unchanged prompts, rubrics, settings, and rules; nothing was refitted.Source: Table 7 · p. 23, Table 6 · p. 23, A Data, collection rounds, rubrics,… · pp. 23–24
• +0.93 points at 41.4% of GPT-6’s fee: the frozen JEV→GPT-6 policy gained 0.93 points on all 1,610 held-out pairs.The headline changed mainly with benchmark composition: JudgeBench, where the policy escalated 65% of pairs, fell from 53% of the earlier held-out set to 17% of the full set; the paper therefore reports the benchmarks separately.Source: A Data, collection rounds, rubrics,… · p. 24

PAIR: response preference

• PAIR asks judges to choose the better of two responses based on correctness, reasoning, instruction following, relevance, and safety, without a tie option.The rubric says not to reward length or confident style and treats candidate text as data rather than judge instructions.Source: PAIR: response preference · p. 24

HALL: evidence-grounded factuality

• HALL labels an answer hallucinated if it contradicts or adds unsupported facts relative to the supplied evidence; otherwise it is supported.The judge evaluates only against the evidence and ignores instructions inside the candidate answer.Source: HALL: evidence-grounded factuality · p. 24

FINAL: final-commitment adjudication

• FINAL judges only the explicitly committed final answer against a trusted reference, allowing equivalent answers and a no_answer label when no answer is committed.Superseded intermediate errors and apology or agreement alone do not change the commitment.Source: FINAL: final-commitment adjudication · p. 24

EVIDENCE: synthetic controls

• EVIDENCE uses supported, contradicted, or unknown labels; its positive-control trajectories all had reference-correct final numbers.The reference is deliberately visible in this adjudication task; all 108 extracted final numbers were correct. Requests contain the state and label probabilities in JSON, without hidden gold labels or source metadata in the inference payload.Source: EVIDENCE: synthetic controls · pp. 24–25

B Judge configurations and measurement

• Judge configurations and measurements specify local-model settings, a frozen 120-decision timing panel, and fees based on billed usage with conservative reservations when usage is missing.Local models have no API-equivalent dollar price; Laya verdicts reproduced across V100 and H100 rescoring, and no candidate was truncated. Fees exclude provider invoices, which were unavailable.Source: B Judge configurations and measurement · pp. 25–26, Table 8 · p. 26

C Human adjudication of judge disagreements

• On JudgeBench, humans sided with GPT-6 on 57/69 disagreements and JEV on 1, yielding a −16.0-point difference ([−20.0, −12.3]) versus −14.6 on labels.By domain, the human favored GPT-6 27–0 in reasoning, 9–0 in coding, 7–0 in math, and 14–1 in knowledge; several items were indecisive.Source: Table 9 · p. 27, C Human adjudication of judge… · pp. 27–28
• The pre-specified audit adjudicated 183 items from 180 source clusters, fully sampling disagreement strata and hash-sampling controls, while hiding labels and judge outputs from the annotator.The primary outcome was the human-adjudicated JEV-minus-GPT-6 difference on JudgeBench; human-corrected accuracy was a sensitivity analysis on adjudicated items only.Source: C Human adjudication of judge… · p. 27
• Final labels were human decisions, but an LLM pass replaced the planned second human pass, and an author adjudicated its 39 disagreements with the annotator.The LLM pass agreed with the human at 58.0% on RewardBench, 83.5% on JudgeBench, and 92.9% on HaluEval; the audit was selective, used team-member annotators, and cannot establish LLM-pass independence.Source: C Human adjudication of judge… · pp. 27–28
• Human-adjudicated differences also favored GPT-6 on RewardBench (−3.0 points, [−5.3, −0.8]) and HaluEval (−2.5 points, [−5.0, −0.4]).Both intervals exclude zero, and humans favored GPT-6 on disputed items more than benchmark labels did by 1.7 and 3.3 task-accuracy points, respectively.Source: C Human adjudication of judge… · p. 28
• On HaluEval, humans decided 24/26 both-wrong items for the judges; decisive label substitutions changed JEV/GPT-6 accuracy to 95.8%/98.3% from 87.5%/86.7% on labels.The label-based +0.8-point near-tie across 240 answers is partly an artifact of label noise near the ceiling.Source: C Human adjudication of judge… · p. 28
• On HaluEval, decisive human-label substitutions reversed JEV’s advantage over GPT-6 on Brier, NLL, and ECE, but this selective-sample sensitivity analysis did not rerun or recalibrate models.Error-detection AUROC is a separate ranking measure, and the correction left only ten JEV errors and four GPT-6 errors.Source: Table 10 · p. 28, C Human adjudication of judge… · p. 28

• Domain rankings change within and across benchmarks; Figure 12 plots main-round hosted accuracy against estimated full-workload fees and timing-panel median latency.Figure 11 gives per-domain base accuracy, except Skywork, which appears in Table 1.Source: D Per-domain results and… · p. 28, Figure 11 · p. 29, Figure 12 · p. 30
• Presentation misleads JEV on RM-Bench when content is held fixed and style varies; the domain-slice comparisons use single base-order calls, not routing’s two-order calls.Prospective workloads are excluded, and Table 11 describes its slice analysis as exploratory, with intervals unadjusted for multiple comparisons.Source: D Per-domain results and… · p. 29, Table 11 · p. 31
• On JudgeBench, JEV trails GPT-6 by 16–19 points for shorter or similarly sized rejected answers and by 4.7 points when the rejected answer is longer.Across length-mismatched pairs, JEV and GPT-6 choose the longer answer in 45.9% and 50.8% of JudgeBench pairs, versus 49.7% with the preferred answer longer; on RewardBench, those figures are 49.3%, 49.4%, and 49.9%.Source: D Per-domain results and… · p. 29, Table 12 · p. 32

E Follow-up benchmark checks

• Three GPT-6 follow-up outcomes were provider content-filter rejections, counted as accuracy errors but excluded from probability diagnostics and not retried or rewritten.The frozen follow-up samples used seeded SHA-256 selection; RewardBench 2 was scored as four-way top-1 accuracy with ties excluded, and RM-Bench used 9,000 judgments across 500 source prompts, styles, and orders.Source: E Follow-up benchmark checks · p. 32
• JEV’s RM-Bench hard-minus-normal change is −8.8 points (95% source-cluster interval [−10.6, −7.1]), showing sensitivity to misleading style.GPT-6’s change is −1.7 points ([−3.2, −0.3]), Skywork’s is −16.8 ([−19.3, −14.4]), and the JEV-minus-GPT-6 difference is −7.1 points ([−8.8, −5.6]); JEV reverses its choice in 381 of 4,500 valid presentation pairs (8.5%), versus GPT-6 in 90 of 4,497 (2.0%). Style effects and reversal instability are distinct; domain and reversal analyses are exploratory, and training overlap is unknown.Source: E Follow-up benchmark checks · pp. 32–33, Table 14 · p. 34
• On RM-Bench, JEV’s q ≥0.9 errors were 1.5% on easy, 3.0% on normal, and 7.9% on hard pairs, with confidence routing least reliable on hard pairs.At q ≥0.99, the bins contained 474 easy, 454 normal, and 354 hard pairs, with 2, 5, and 5 errors, respectively; treating each presentation order as a separate judgment, JEV’s error rates were 1.8%, 3.9%, and 10.0%, versus GPT-6’s 3.9%, 3.6%, and 4.9%.Source: E Follow-up benchmark checks · p. 33
• On 100 RewardBench 2 prompts, JEV scored 73% and GPT-6 75%; escalating JEV’s below-0.5-confidence 20% matched GPT-6, while escalating the below-0.7 40% reached 77%.These thresholds were read from the same prompts, so a two-point difference represents two prompts.Source: E Follow-up benchmark checks · p. 33

F Answer format and gold-blind extraction

• Gold-blind extraction identifies only the final committed option without judging correctness or seeing reference metadata, then code maps the option to correct or incorrect.Later explicit revisions supersede earlier answers; unresolved alternatives and missing commitments remain distinct as ambiguous and no_answer outcomes.Source: F Answer format and gold-blind… · pp. 35–37
• For natural replies, extraction produced more ambiguous outputs and an inspectable answer but no agreement gain over direct adjudication.The four-way contract adds an ambiguous outcome to the original three-way labels, so these scores are not comparable with Table 1’s 94.0%; without independent option-level gold, the comparison measures downstream agreement, not extraction accuracy.Source: F Answer format and gold-blind… · p. 35
• In constructed controls, GPT-4.1 mini’s errors mostly fell at the ambiguous/no-answer boundary, while JEV’s free-response disagreements concentrated on supplied wrong-answer conditions.The controls use inherited HaluEval labels and matched multiple-choice and free-response formats; examples show semantic adjudication can diverge from lexical option equality, so the authors do not treat every disagreement as a model error.Source: Table 16 · p. 35, F Answer format and gold-blind… · pp. 35–36
• The task-type follow-up used 240 HaluEval answers from 120 source questions, and its short-answer versus prose split is descriptive rather than a general test of free-response correctness.An initial 440-outcome JEV trial was excluded after an accounting write-lock failure before accuracy was inspected; interrupted records supplied no task decisions or imputed latency.Source: F Answer format and gold-blind… · p. 36

G Natural free-response evaluation

• Every judge agreed poorly with human labels on balanced reference-free responses, despite high mean confidence; the authors recommend independent verification of consequential factual claims.The paper cautions that this result does not show a generative interface would resolve the problem.Source: G Natural free-response evaluation · p. 37, Figure 16 · p. 38

H Output validity and additional metrics

• Validity analyses separate provider generation failures, contract-invalid HTTP-200s, and transport or other HTTP failures; primary accuracy counts failures as wrong, while probability metrics use valid responses.This keeps service availability distinct from reasoning quality.Source: H Output validity and additional metrics · p. 38, Table 19 · p. 38, Table 20 · p. 39

I Stability and interface diagnostics

• Native-confidence error-detection AUROC was 0.876, 0.745, and 0.842 on RewardBench, JudgeBench, and HaluEval; primitive differences averaged 0.015 for Choice–Score and 0.011 for joint versus standalone Choice.The Noul complement residual reaches 0.19; the provider does not publish its native-confidence implementation.Source: I Stability and interface diagnostics · pp. 40–41

J Confidence under presentation order: details

• JEV’s accuracy gain from averaging two orders was not significant: +0.76 points on 1,850 pairs, +0.62 on 1,610 held-out pairs, and +2.04 on held-out JudgeBench pairs.Only Gemini 3 Flash gained significantly after correcting for 18 judges; the authors characterize averaging as a robustness step rather than a source of accuracy.Source: J Confidence under presentation order:… · p. 43, Table 25 · p. 43

K Probability quality and selective prediction

• Twelve frozen two-order JEV policies were selected on 96 pilot pairs and evaluated offline on 1,610 held-out preference pairs.The policies are reported with cascade-minus-fallback accuracy differences and fee ratios.Source: K Probability quality and selective… · p. 46, Table 29 · p. 46
• On 1,340 RewardBench pairs, alternatives ranged from 98.9% (Laya Typed-decisions) to 103.1% (Skywork) of GPT-6 accuracy; on 270 JudgeBench pairs, JEV and several alternatives reached 99.2–99.6%.This was a retrospective reuse of the selection rule, not a pre-specified test of the added first stages; the two-point selection tolerance is not a held-out guarantee.Source: K Probability quality and selective… · p. 47
• A low share of confident errors means most errors occur below q = 0.9, where a gate can escalate them; confident-error counts also depend on how often a judge accepts.The analysis uses base-order judgments under supplied labels, and HaluEval shares are dominated by label noise.Source: K Probability quality and selective… · p. 47, Table 30 · p. 47
• As selection labels rose from k = 96 to 384, median escalation fell from 32% to 13% while risk fell from 5.2% to 2.9%; workload-specific thresholds helped because uniform draws underrepresented JudgeBench.The simulation reused one benchmark pool and did not model distribution shift.Source: K Probability quality and selective… · pp. 47–48, Table 31 · p. 48

L Open-weight typed decisions: Laya

• Gold-blind extraction beat direct adjudication by 22.9 points for English and 36.2 for Typed-decisions; Typed-decisions gained 9.3 points on natural replies, while both scored higher on free-response controls.The natural-reply result used 150 replies; extraction reduces the task to matching an option rather than judging it.Source: L Open-weight typed decisions: Laya · pp. 48–49
• Laya’s checkpoints returned valid, repeatable probabilities, but preference-pair accuracy was 47–50% on RewardBench and 49% on JudgeBench, while four-way selection scored 20–23% against 25% chance.Confidence was uninformative about errors (AUROC 0.48–0.53), and the frozen rule accepted only 3–5% of held-out pairs and none of JudgeBench.Source: L Open-weight typed decisions: Laya · p. 48, Table 32 · p. 49

M Prospective live test on PPE and JudgeBench

• JEV alone scored 78.2% on PPE and 70.9% on JudgeBench, versus GPT-6’s 88.2% and 94.7%; the cascade matched GPT-6 on seen and unseen JudgeBench questions.The frozen prospective policy escalated held-out pairs when either JEV order was invalid or below its workload threshold.Source: M Prospective live test on PPE and… · pp. 49–50, Table 34 · p. 50

N Live replication of the frozen policy

• The live replication differed from offline outcomes by at most 0.4 points, with JEV and GPT-6 verdicts each matching their main-round counterparts on 99.0% of pairs.Median latency was 0.18 seconds for accepted pairs and 2.50 seconds for escalated pairs; the run cost $8.81.Source: Table 36 · p. 51, N Live replication of the frozen policy · p. 51

O Illustrative cases and rubric alignment

• Prioritizing final-answer correctness in a JudgeBench rubric ablation moved base accuracy from 78.6% to 79.4% but left pooled two-order accuracy at 79.6%.Illustrative disagreements involved a correct final choice with a flawed derivation, an ambiguous hallucination label, and an explicit output constraint—not a general verbosity bias.Source: O Illustrative cases and rubric… · p. 52

P Supplementary reproducibility package

• The supplementary package includes sampled inputs and rubrics, retained decisions and fee records, scripts for principal analyses, and materials for human adjudication, Laya, prospective tests, and live replication.It excludes private-source records, raw API logs, account metadata, and credentials; analyses require neither API calls nor a GPU, and no model is trained or fine-tuned.Source: P Supplementary reproducibility package · p. 52