JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

1 Introduction

• LLM judges scale contextual assessment across open-ended tasks by applying natural-language rubrics, unlike fixed metrics that struggle to capture varied notions of quality.Figure 1 places the human, generative, and decision-only interfaces alongside one another.Source: §1 · p. 1, Figure 1 · p. 1
• Reasoning-intensive judgments can raise fees and latency, while simple problems may receive extended computation with little additional benefit.Repeated evaluation across responses and model revisions makes judge cost part of system development and operation.Source: §1 · p. 2
• LLM-reported confidence can be overconfident, so its reliability must be tested for the model, task, and elicitation procedure.Confidence quality matters both for trusting judgments and allocating evaluation resources.Source: §1 · p. 2
• The study asks whether JEV’s typed verdict and label probabilities can provide a low-cost first pass and flag inputs for stronger evaluation.It evaluates accuracy, probability quality, and operating cost together; a typed probability output alone does not establish calibration.Source: §1 · p. 2
• The comparison spans preference, factuality, and answer-adjudication tasks, with fees, latency, and blinded human adjudication used to assess disagreements with the strongest judge.Generative baselines also receive a decision-and-probability output contract without a rationale; the comparison is between complete configurations because JEV is proprietary.Source: §1 · p. 2
• JEV is competitive on ordinary preference and evidence-grounded factuality at substantially lower measured cost, but difficult correctness and misleading answer style expose clear limits; confidence can support cascades that escalate uncertain decisions.Source: §1 · p. 2

2 Related work

• The study fixes the verdict consumed by a pipeline and measures reversal, paraphrase, and style sensitivity, against prior work on position, verbosity, self-preference, presentation effects, and rubric variation.Related work includes balanced-order evaluation (Wang et al., 2024), rubric wording variation (Bagaria et al., 2026), and broader evaluation taxonomies (Li et al., 2025).Source: §2 · p. 2
• The evaluation draws on preference and correctness benchmarks, evidence-grounded hallucination labels, harder multi-response selection, and a modern reward model beyond older rankers.The passage names PairRM (Jiang et al., 2023), Skywork-Reward-V2 (Liu et al., 2026), RewardBench (Lambert et al., 2025), JudgeBench (Tan et al., 2025), HaluEval (Li et al., 2023), RewardBench 2 (Malik et al., 2026), and RM-Bench (Liu et al., 2025).Source: §2 · p. 2
• Prior confidence calibration, selective classification, and model cascades motivate economical escalation; this study contributes measured fees and frozen thresholds, not a new routing algorithm.Closest cited approaches cascade judges with human-agreement guarantees (Jung et al., 2025) or route uncertain reward-model comparisons to a strong LLM judge (Xu et al., 2025).Source: §2 · pp. 2–3

3 Tasks and experimental protocol

• Tasks cover pairwise prose ranking, single-answer assessment against supplied evidence, and final multiple-choice or numeric-answer adjudication; a typed judge output alone does not solve closed-answer tasks.A judge returning only a label can still assess free text, so evaluated answer format and judge output are treated separately.Source: §3 · p. 3
• The public sample includes 400 RewardBench pairs, 350 JudgeBench GPT-4o items, and 240 evidence-grounded HaluEval judgments, with supplied benchmark labels retained.The RewardBench sample is balanced across four categories but does not reproduce official subset weighting; benchmark labels have mixed origins.Source: §3 · p. 3
• Existing data add 150 final-answer adjudications, 108 reference-correct replies from GSM8K conversations, and 64 synthetic evidence judgments.The adjudication labels are existing-label agreement scores because annotator identity is unavailable; the controls test elementary handling.Source: §3 · p. 3
• The 642-item pilot was frozen before inference, with source-question splits for threshold fitting and pilot testing; a 670-item extension was frozen before pilot accuracy was inspected.The later multi-family comparison is exploratory because the extension was evaluated after pilot results were known.Source: §3 · p. 3
• Each judge receives 894 diagnostics, including all 750 preference pairs in reversed order and repeated or paraphrased-rubric requests on 48 fixed examples.PairRM covers both orders of the 750 pairs, while the three earlier GPT baselines have matching retained data.Source: §3 · p. 3

4 Judges and measurement

• Choice returns probabilities over specified labels, Noul returns a yes-probability, and Score returns probabilities over ordered rubric levels; primary experiments use one Choice question per request.At collection time, JEV 1.13.0 cost $0.042 per million input tokens with no output-token charge.Source: §4 · p. 3
• JEV’s native confidence correlates with maximum label probability at 0.971, 0.999, and 0.948 on RewardBench, JudgeBench, and HaluEval, respectively; the study uses q = max_k p_k for confidence analyses.Native confidence is retained for interface diagnostics.Source: §4 · p. 3
• The comparison includes thirteen hosted judges and four local baselines; PairRM has a shorter context and no rubric, while Skywork scores each response separately.PairRM truncation affects seven RewardBench and 59 JudgeBench pairs; its untruncated accuracies are 67.7% and 53.3%, respectively.Source: §4 · pp. 3–4
• Generative judges receive the same instructions and state as JEV and return decisions with label probabilities but no rationale; their reasoning settings vary and measured fees and latency absorb those differences.The study compares complete configurations rather than compute-matched architectures.Source: §4 · p. 4
• Outputs are checked for schema validity, label membership, finite normalized probabilities, and verdict–argmax consistency; invalid outcomes count as accuracy errors.Probability metrics use valid outputs, transport failures receive at most three attempts, and nothing is repaired or rerun for a better answer.Source: §4 · p. 4
• Accuracy, macro-F1, Brier score, clipped NLL, ECE, and error-detection AUROC are reported, with paired differences assessed by source-question cluster bootstrap resampling.Latency comes from a frozen 120-decision panel; fees use reported usage at collection-time prices or conservative reservations when usage is missing, and are estimates rather than invoices.Source: §4 · p. 4

• JEV scores 92.2% versus GPT-6’s 93.5% on RewardBench and 87.5% versus 86.7% on HaluEval; blinded human adjudication favors GPT-6 on both.On HaluEval, 24 of 26 items both judges missed had labels unsupported by evidence; correcting those labels gives JEV 95.8% and GPT-6 98.3%.Source: §5 · p. 4
• JEV scores 78.6% versus GPT-5.6 and GPT-6’s 93.1% on JudgeBench, a paired difference of −14.6 points; human adjudication finds a −16.0-point difference.The gap is widest in reasoning and coding, and adjudication sides with GPT-6 on 57 of 69 disputed items.Source: §5 · p. 5
• On the 150 existing replies, JEV reaches 94.0% accuracy and 0.923 macro-F1, while saturated controls discriminate little among judges.The reply scores are agreement with existing labels; annotator identity is unavailable.Source: §5 · p. 5
• On RM-Bench, JEV falls from 84.0% on same-style pairs to 74.8% when the rejected answer is more elaborately written, while GPT-6 rises from 93.3% to 94.6%.On four-way selection, JEV scores 73.0% versus GPT-6’s 75.0%; choosing correctly and resisting misleading style are distinct abilities.Source: §5 · p. 5
• On reference-free natural prose, JEV scores 52.5% and remains confident, with mean maximum probability 0.90; all three tested judges are near chance.The two prose samples differ in content and label provenance, so the results indicate workload dependence rather than a pure format effect.Source: §5 · p. 6
• JEV returns a valid judgment on every base item, while some generative configurations fail at the provider, contract, or transport stage.Failed calls count as no judgment, not as reasoning errors.Source: §5 · p. 6
• JEV’s median latency and fee are 0.152 seconds and $0.044 per 1,000 judgments, versus GPT-6’s 1.885 seconds and $12.182.That is about 277 times cheaper on this workload.Source: §5 · p. 6
• Within three points on ordinary preference and evidence-grounded factuality at one to two orders of magnitude lower fee, JEV trails GPT-6 by 9–20 points on derivation or style-adversarial judgments.Table 2 summarizes paired differences by workload; its guidance is descriptive.Source: §5 · p. 6

• JEV changes no decision across 96 repeated comparisons but changes four across 48 paraphrases; reversing candidate order changes 3.25% of RewardBench and 11.14% of JudgeBench decisions.JEV picks the first position 48.9% and 48.4% of the time, so the inconsistency is not a positional preference.Source: §6 · p. 6
• JEV’s Brier scores are 0.111, 0.297, and 0.176 on RewardBench, JudgeBench, and HaluEval; GPT-6’s are 0.104, 0.095, and 0.245.GPT-6 is better calibrated on the hard comparisons, while JEV’s probabilities are better on evidence-grounded uncertainty.Source: §6 · p. 6
• At q ≥0.9 on HaluEval, JEV makes 10 errors in 199 judgments and GPT-6 makes 28 in 233, while their coverage differs.Their error-detection AUROC values across RewardBench, JudgeBench, and HaluEval are 0.869, 0.745, 0.863 for JEV and 0.891, 0.907, 0.899 for GPT-6.Source: §6 · p. 6
• Temperature scaling improves JEV’s extension NLL on HaluEval from 0.333 to 0.284 but worsens it on JudgeBench and RewardBench.The fitted temperatures differ across workloads, and the authors conclude that each workload needs its own validation.Source: §6 · p. 7
• Aligned Choice, Noul, and binary Score probabilities differ by a mean Choice–Noul gap of 0.055, while hard labels disagree once in 48 audited examples.Co-question context, stochastic variation, and rounding contribute to the differences.Source: §6 · p. 7

7 Error complementarity and deferral

• The gate accepts JEV’s candidate verdict when confident and otherwise defers to a stronger judge; pairwise probabilities are aligned and averaged across both candidate orders.The threshold is selected per workload; invalid first-stage outputs always defer.Source: §7 · pp. 8–9, Figure 4 · p. 9
• At τ=0.9, RewardBench reaches 94.0% versus GPT-6’s 93.5% at 22% of its fee; JudgeBench retains 98.2% at 62% of the fee.On HaluEval, thresholds do not move label-based accuracy, but under human-corrected labels τ=0.8 matches GPT-6’s 98.3% at 13% of its fee.Source: §7 · pp. 8–9
• At τ=0.9, the frozen GPT-6 cascade accepts 53.7% of 510 extension pairs, scores 92.5% versus 93.1%, and uses 56.8% of the fallback fee.The paired change is −0.59 points, with a 95% interval of [−1.78, 0.59].Source: Table 3 · p. 9, §7 · p. 9
• On hard RM-Bench pairs q’s correctness AUROC falls to 0.770; on reference-free prose it is 0.518 and no threshold helps.For hard pairs, a τ=0.9 cascade retains 96.5% of GPT-6 accuracy; reaching 98.7% requires τ=0.95 and 58% escalation.Source: §7 · pp. 9–10
• At τ=0.9, the pooled single-order cascade reaches 91.3% versus GPT-6’s 91.7% at 47% of its fee, retaining 99.6% accuracy.Random escalation at the same 34% budget reaches 88.1%, while the label-aware oracle reaches 94.4%.Source: §7 · p. 8, Figure 5 · p. 10
• JEV accuracy rises with q; on items accepted at q ≥0.9 it scores 95.8% versus GPT-6’s 96.5%, while GPT-6 leads by 15 points on deferred items.The confidence pattern holds within each of the three tasks.Source: §7 · p. 8, Figure 5 · p. 10

8 Conclusion

• JEV stays within three points of the strongest tested judge on ordinary preference, evidence-grounded factuality, and final-answer adjudication, at about $0.04 per 1,000 judgments.The conclusion reports 0.15 seconds per judgment and says JEV returns a valid verdict on every item.Source: §8 · p. 10
• Escalate judgments requiring derivation checks or resistance to elaborately written wrong answers, and do not trust tested judges to grade natural prose without a reference.The conclusion advises treating confidence as an escalation signal, not a certificate.Source: §8 · p. 10
• Validate escalation thresholds on local selection data and held-out items, and check temperature per workload before extending the approach to a new workload.The conclusion also recommends judging preference pairs in both orders and counting invalid outputs as errors.Source: §8 · p. 10

Limitations

• Only one proprietary JEV version and a chosen set of configurations were evaluated; compute, model, and serving differences prevent an architecture-matched comparison.Training overlap is unknown, and some service aliases or preview models can change.Source: Limitations · p. 10
• Decision-only prompts omit generated explanations, explanation faithfulness, extended deliberation, and many grading rubrics.The comparison therefore does not cover those aspects of judging.Source: Limitations · p. 10
• Multi-family analyses are exploratory, and the extension shares benchmark families and was not newly hidden from the analyst.Small calibration and diagnostic sets yield uncertain estimates; descriptive ECE and many comparisons do not support isolated superiority claims.Source: Limitations · p. 11
• Existing annotations have incomplete provenance, and the 183-item human adjudication is a disagreement-selected author audit, not a random or large-scale consensus audit.RewardBench preferences remain partly subjective (human–LLM κ = 0.29), and 36 adjudicated labels are indecisive; specialized professional domains are untested.Source: Limitations · p. 11
• Cascade fees are simulated, and the single-order cascades are post hoc; only frozen two-order policies test a prespecified rule.Latency reflects one client location and collection window, and local GPUs have no API-equivalent prices.Source: Limitations · p. 11

Ethical considerations

• The study reuses benchmark responses and existing annotations; its only new human judgments are 183 blinded adjudications by research-team members on public items.The authors say provenance and licensing need review before public redistribution, and warn against using any automated judge as sole arbiter of consequential decisions.Source: Ethical considerations · p. 11

A Data partitions, rubrics, and request examples

• The data section defines separate task rubrics and keeps repeated answers to a source question together across partitions, excluding extension labels from temperature and threshold fitting.The existing-response sample contains 150 replies from six models across 119 source-question clusters; its judge sees the question, reference, and reply, not conversation history.Source: A Data partitions, rubrics, and… · pp. 14–15, Table 4 · p. 14

B Model configurations and accounting

• JEV is charged for input tokens and has zero output-token fees; local models have no assigned API-equivalent dollar price.The study documents requested model identifiers, pricing, and serving settings; fees use documented cache rules and conservative reservations when usage is missing.Source: B Model configurations and accounting · p. 15, Table 5 · p. 16

C Operational validity and additional metrics

• Operational reporting separates provider generation failures, invalid contract responses, and transport failures so service availability is not conflated with reasoning quality.Primary accuracy counts failures as wrong, while probability metrics condition on valid responses; requests omit hidden labels and source metadata.Source: C Operational validity and additional… · p. 16, Table 6 · p. 16, Table 7 · p. 17

D Domain results and quality–efficiency trade-offs

• Domain rankings change within and across benchmarks; the domain and quality–efficiency figures report accuracy by domain and fresh hosted quality against estimated fees and isolated timing.The figure captions specify source-cluster accuracy intervals and the workload sizes used for fee and timing estimates.Source: D Domain results and… · p. 17, Figure 7 · p. 17, Figure 8 · p. 18

E Follow-up benchmark checks

• GPT-6 had three invalid outcomes, all provider content-filter rejections, which remain in the accuracy denominator but are omitted from probability diagnostics.RewardBench 2 uses four-way top-1 selection and excludes ties; its score is not comparable with binary RewardBench 1 accuracy.Source: E Follow-up benchmark checks · p. 19, Table 8 · p. 19
• JEV’s RM-Bench hard-minus-normal change is -9.2 points; GPT-6’s is +1.3 and Skywork’s is -20.6, while JEV’s change differs from GPT-6’s by -10.4 points.On RM-Bench, JEV reverses its semantic choice in 63 of 720 valid pairs (8.75%), versus GPT-6 in 11 of 717 (1.53%).Source: E Follow-up benchmark checks · pp. 19–20, Table 9 · p. 20

F Probability quality and selective prediction

• The cascade tables distinguish frozen two-order policies from single-order confidence cascades whose thresholds were not frozen.The single-order table defines acceptance by JEV maximum label probability and reports retained accuracy and fee ratio relative to GPT-6 alone.Source: F Probability quality and selective… · p. 21, Table 13 · p. 23

G Stability and interface diagnostics

• Native confidence is a provider-computed distribution statistic; its error-detection AUROC is 0.870 on RewardBench, 0.745 on JudgeBench, and 0.865 on HaluEval.The stability analyses distinguish repeats, rubric paraphrases, and response reversal, separating aggregate position preference from inconsistency.Source: G Stability and interface diagnostics · p. 24, Table 15 · p. 24

H Illustrative cases and rubric alignment

• A JudgeBench parachutist answer exposes a target mismatch: JEV favors the answer with a flawed derivation, while GPT-6 favors the correct final choice.The rubric-priority ablation moves JEV base accuracy from 78.6% to 79.4% (paired change +0.86 points, [-1.14, 3.14]) and leaves order-averaged accuracy at 79.6%.Source: H Illustrative cases and rubric… · p. 26
• On HaluEval item 9045, both blinded adjudication passes judged the abbreviated answer hallucinated, matching JEV rather than GPT-6.The evidence named no agent for the restoration, although the benchmark label treated “President López” as hallucinated.Source: H Illustrative cases and rubric… · p. 26
• A poem-continuation case is an instruction-following disagreement, not evidence of a general verbosity bias.JEV favors a six-line continuation, while GPT-6 and both adjudication passes favor the benchmark’s one-line answer.Source: H Illustrative cases and rubric… · p. 26

I Human adjudication of judge disagreements

• −16.0 points ([−20.0, −12.3]) was the human-adjudicated JEV-minus-GPT-6 difference on JudgeBench, versus −14.6 on benchmark labels.Humans sided with GPT-6 on 57 of 69 disagreements and with JEV on one; disputed-item adjudication also favored GPT-6 on RewardBench and HaluEval.Source: Table 16 · p. 26, I Human adjudication of judge… · p. 27
• 24 of 26 HaluEval items both judges missed were decided for the judges, while JudgeBench’s both-wrong stratum contained 10 of 15 indecisive cases.Correcting HaluEval labels on adjudicated items yielded 95.8% for JEV and 98.3% for GPT-6, versus 87.5% and 86.7% on labels.Source: I Human adjudication of judge… · p. 27
• The audit is not a random benchmark audit: one human pass plus an LLM pass replaced two human passes, and the adjudicator was an author.Benchmark labels remain unchanged in every primary result; the LLM rater may share failure modes with the judges, so its agreement does not establish independence.Source: I Human adjudication of judge… · p. 27

J Answer format and gold-blind extraction

• Answer-format controls separate gold-blind extraction from grading: extraction sees the question and reply, then code maps the extracted option against the trusted reference.The controls use explicit commitment rules, including treating a later revision as final; seeded samples pair correct and wrong answers across constructed formats.Source: J Answer format and gold-blind… · pp. 28–29
• Natural option-extraction results measure downstream label agreement, not extraction accuracy, because there is no independent option-level gold.Five retained questions lack complete options, and the natural four-way rubric differs from the primary three-way evaluation.Source: J Answer format and gold-blind… · pp. 28–29
• The constructed-control disagreements distinguish extraction from grading, with JEV’s free-response disagreements concentrated on supplied wrong-answer conditions.Those conditions inherit HaluEval labels, and lexical option equality differs from semantic adjudication; the transparent controls do not establish performance on unrestricted natural responses.Source: J Answer format and gold-blind… · p. 28, Figure 17 · p. 28

K Natural free-response evaluation

• The natural-prose follow-up uses 80 document-grounded summaries and 80 balanced reference-free responses, measuring agreement with existing annotations rather than natural hallucination prevalence.The reference-free inputs contain no evidence or labels, and the balanced sample uses labels only for sampling.Source: K Natural free-response evaluation · p. 30, Table 19 · p. 30
• All three judges agree poorly with human labels on the balanced reference-free sample, despite high mean confidence; the two workloads do not isolate the effect of supplied evidence.The study recommends independent verification of consequential factual claims but does not show that a generative interface would resolve the problem.Source: Figure 18 · p. 30, K Natural free-response evaluation · p. 30

L Supplementary reproducibility package

• The supplementary package includes public-task inputs, outcomes, metric tables, reproduction code, and human-adjudication materials; reproducing public results needs neither API calls nor a GPU.Original API logs and private-source response records remain in the internal artifact.Source: L Supplementary reproducibility package · p. 31