Source-linked AI summary
From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks
Zheng Zhang, Lufei Li, Xinyue Tan, Yuanhao Zeng, Ziwei Shan, Yexin Li, Kan Ren
TL;DR
Open-ended tasks lack verifiable ground-truth answers, and prior work has focused on using Judge judgments for policy training while giving limited attention to their intrinsic quality or test-time utility. This paper varies Judge protocols and studies their use in training and test-time inference. It finds that judgment quality and downstream utility do not always align, while test-time Judge guidance improves responses with gains that vary across strategies.
Problem
Prior work largely converts Judge judgments into training rewards, with limited attention to intrinsic judgment quality or whether Judges can directly improve responses during test-time inference.
Source: §1 · pp. 1–2Method
The paper varies verdict granularity, critique usage, and evaluation batching, and evaluates Judge use in policy training and test-time Best-of-N selection, revision, and beam search.
Source: §1 · p. 2Results
Takeaways & Limitations
LLM Judges on open-ended tasks should be evaluated for both intrinsic judgment quality and downstream utility.
Source: §9 · p. 9
Abstract
from arXiv · showhide
LLM-as-a-Judge is increasingly used to evaluate policy responses on open-ended tasks that lack ground-truth answers. Existing work often directly converts the resulting judgments into reward signals for policy training, paying limited attention to intrinsic judgment quality and largely restricting the use of Judges to training-time supervision. We systematically investigate judgment quality and downstream utility by examining both how judgments are elicited and how they are used. For judgment elicitation, we vary the Judge protocol along three dimensions: verdict granularity, critique usage, and evaluation batching. For judgment usage, beyond policy training, we extend Judge to test-time inference through Best-of-N selection, Judge-guided revision, and beam search. We find that, (i) Surprisingly, judgment quality and downstream utility do not always align. (ii) Judge protocol design substantially affects both intrinsic judgment quality and downstream utility. (iii) Judge guidance effectively converts test-time compute into performance gains, with benefits varying across inference strategies. Our results call for a multifaceted evaluation of LLM Judges on open-ended tasks, encompassing intrinsic judgment quality, and downstream utility.
1 INTRODUCTION
LLM Judges provide rubric-based evaluation signals for open-ended tasks, but prior work has focused mainly on using them for policy training, with limited attention to judgment quality or test-time utility. This paper examines how Judge protocols shape judgment quality and downstream utility across training and inference.
1 INTRODUCTION
- Judgment quality and downstream utility do not always align, so policy performance alone may not reflect the quality of the judgments that guide it.Prior work primarily studies downstream policy performance, while the relationship between that performance and underlying judgment quality has not been systematically examined.Source: §1 · pp. 1–2
- The study varies verdict granularity, critique usage, and evaluation batching, measuring judgment accuracy, critique–verdict consistency, and stability.It also tests Best-of-N selection, Judge-guided revision, and beam search on open-ended medical and scientific question-answering tasks.Source: §1 · p. 2
- Best-of-N selection, Judge-guided revision, and beam search improve policy responses, with gains varying across strategies.Judge protocols also substantially influence judgment quality and policy-optimization outcomes.Source: §1 · p. 2
2 RELATED WORK
Prior work uses rubric-based LLM judgments to support policy optimization and studies Judge behavior in preference or rubric-evaluation settings. The paper identifies intrinsic judgment quality and utility beyond policy training as areas receiving limited attention.
2 RELATED WORK
- Studies use rubric-based Judge rewards to optimize policies for open-ended question-answering, including medical and scientific tasks.Examples include training on factual soundness and writing quality, and optimizing medical dialogue policies with rubric-based rewards.Source: §2 · p. 2
- Pairwise analyses examine Judges as preference evaluators and how protocols affect preference alignment, while pointwise work studies rubric-based response evaluation.Pointwise examples investigate rubric design and rubric provenance.Source: §2 · pp. 2–3
- Prior analyses focus on LLM alignment or rubric design, while intrinsic judgment quality in open-ended tasks and Judge utility beyond policy training receive limited attention.The paper frames its focus against work that primarily converts rubric-based judgments into policy-training rewards.Source: §2 · pp. 2–3
3 PRELIMINARY
The preliminary section describes rubric-based Judge rewards for policy optimization and organizes Judge protocols along three dimensions. These definitions establish the reference configuration for the paper’s analysis.
3.1 JUDGES AS TRAINING SIGNAL PROVIDERS FOR OPEN-ENDED TASKS
- Judges evaluate policy responses against query-specific rubric criteria, whose importance weights contribute to a normalized reward usable by RL algorithms such as GRPO.Each criterion receives a structured prompt and a binary verdict indicating whether the response satisfies it.Source: §3.1 · p. 3
3.2 JUDGE PROTOCOLS
- Verdict granularity contrasts binary True/False judgments with 0–10 ratings that express the degree to which a response satisfies a criterion.The paper treats these as alternative formats for expressing criterion-level judgments.Source: §3.2 · p. 3
- Critique usage distinguishes critique-before-verdict, verdict-before-critique, and verdict-only protocols.The dimension captures whether the Judge produces a natural-language critique and where it appears relative to the verdict.Source: §3.2 · p. 3
- Evaluation batching compares separate Judge calls for each rubric criterion with one call that assesses all criteria.The reference configuration is T/F, Critique+Verdict, and Independent Evaluation; the study varies one protocol dimension at a time from that setting.Source: §3.2 · p. 3
4 ANALYTICAL FRAMEWORK
The analytical framework evaluates both RL-training utility and intrinsic judgment quality. It measures reward resolution and downstream task performance alongside accuracy, critique–verdict consistency, and stability.
4 ANALYTICAL FRAMEWORK
- The framework assesses Judge protocols through training utility and intrinsic judgment quality, using downstream policy performance and judgment metrics, respectively.Judgment metrics include accuracy, critique–verdict consistency, and stability.Source: §4 · pp. 3–4
4.1 DOWNSTREAM UTILITY FOR RL TRAINING.
- Reward resolution is measured by response-level reward ties and groups receiving uniform rewards; lower rates indicate more distinctions or more groups with learning signals.The framework also evaluates trained policies on medical and scientific test sets using an external grader.Source: §4.1 · p. 4
- Training experiments use GRPO on medical and scientific tasks, with Qwen policies and either Qwen2.5-3B-Instruct or GPT-OSS-20B as training-time Judge.GPT-OSS-120B serves solely as the external evaluator of rubric-based test scores.Source: §4.1 · p. 4
4.2 JUDGMENT QUALITY.
- Judgment accuracy is assessed against majority-vote gold labels from GPT-OSS-120B, GPT-4.1, and DeepSeek-V3.2, using accuracy, TPR, and TNR.The metrics track overall label agreement and the identification of positive and negative cases.Source: §4.2 · p. 4
- Judgment quality also includes critique–verdict consistency and stability, measured by repeated-call agreement and robustness across conservative, neutral, and permissive decision rules.A separate GPT-OSS-120B auditor assesses whether critiques support their verdicts.Source: §4.2 · pp. 4–5
5 VERDICT GRANULARITY
The section compares binary T/F verdicts with 0–10 Rating verdicts, finding that Rating generally improves downstream policy performance despite no consistent advantage in intrinsic judgment quality.
5 VERDICT GRANULARITY
- The experiment compares T/F verdicts with Rating scores from 0 to 10 while holding other protocol components and training configurations fixed.Source: §5 · p. 5
5.1 DOWNSTREAM UTILITY IN RL TRAINING
- Rating achieves a higher mean test score than T/F in 10 of 16 matched comparisons, with GPT-OSS-20B as Judge favoring Rating in seven of eight.Rating also reduces Response-Level Reward Tie Rate in all eight settings and Group-Level Reward Uniformity Rate in seven, providing more informative optimization signals.Source: §5.1 · p. 5, Table 1 · p. 5
5.2 JUDGMENT QUALITY
- Rating supports more effective policy training than T/F judgments despite showing no consistent advantage in intrinsic judgment quality.Source: §5.2 · p. 5
6 CRITIQUE USAGE
The section examines how critique placement affects policy training and intrinsic judgment quality. Verdict+Critique generally performs best downstream and offers the strongest overall quality balance.
6 CRITIQUE USAGE
- The critique-usage experiment compares Critique+Verdict, Verdict Only, and Verdict+Critique while holding other protocol components and training configurations fixed.Source: §6 · p. 5
6.1 DOWNSTREAM UTILITY IN RL TRAINING
- Verdict+Critique achieves the highest mean policy performance in 12 of 16 settings, compared with three each for Critique+Verdict and Verdict Only.Its advantage is strongest with Qwen2.5-3B-Instruct; no critique protocol consistently improves reward resolution.Source: §6.1 · pp. 5–6, Table 2 · p. 6
6.2 JUDGMENT QUALITY
- Verdict+Critique provides the best overall balance across judgment quality dimensions while yielding the strongest downstream policy performance.It has the highest Accuracy in five of eight settings, outperforms Critique+Verdict in consistency in seven of eight, and achieves the highest Decision-Rule Robustness in seven of eight.Source: §6.2 · p. 6
7 EVALUATION BATCHING
The section compares independent and batched rubric evaluation, finding that batching trades lower evaluation cost for generally weaker judgment quality and downstream utility. Its effects include more reward ties and position-dependent shifts, especially for the smaller Judge.
7 EVALUATION BATCHING
- The evaluation compares Independent calls, which assess criteria separately, with Batched calls, which assess all criteria in one call.The section examines both downstream training utility and intrinsic judgment quality.Source: §7 · p. 7
7.1 DOWNSTREAM UTILITY IN RL TRAINING
- Batched evaluation yields higher mean performance in only six of 16 matched comparisons, while increasing response-level reward ties in seven of eight settings and group-level reward uniformity in all eight.These changes reduce response-level discrimination and leave fewer groups with non-zero relative advantages for GRPO.Source: §7.1 · p. 7, Table 3 · p. 7
7.2 JUDGMENT QUALITY
- Batched evaluation reduces judgment accuracy in seven of eight settings and lowers repeated-call agreement in seven of eight and decision-rule robustness in six.Its effects on critique–verdict consistency are especially negative for Qwen2.5-3B-Instruct, while changes are mixed with GPT-OSS-20B.Source: §7.2 · p. 8
- For Qwen2.5-3B-Instruct, batching increases the True-Verdict Rate at every rubric position, with larger increases early and midway than near the end.GPT-OSS-20B shows much smaller changes, indicating less sensitivity to batching.Source: Figure 6 · p. 7, §7.2 · p. 8
8 DOWNSTREAM UTILITY FOR TEST-TIME INFERENCE
The paper tests whether Judges can improve policy responses at inference time without updating policy parameters, using Best-of-N selection, revision, and beam search. Selection and revision improve responses consistently in the reported settings, while beam-search gains depend on policy and domain.
8 DOWNSTREAM UTILITY FOR TEST-TIME INFERENCE
- The test-time approaches are Best-of-N selection among complete responses, feedback-driven revision of a response, and Judge-scored beam search over partial responses.The policy parameters remain unchanged during this inference-time guidance.Source: §8 · p. 8
8.1 BEST-OF-N
- Judge-guided Best-of-N selection beats Direct in all 32 settings, with its mean gain rising from 7.8 points at N = 4 to 17.2 points at N = 40.Average-of-N remains close to Direct as N increases.Source: §8.1 · p. 8
8.2 JUDGE-GUIDED REVISION
- A single Judge-guided revision improves the mean test-grader score over Direct in all eight settings, by 1.1 to 22.8 points.Qwen3-30B-A3B yields larger gains than GPT-OSS-20B in every matched comparison.Source: §8.2 · p. 9, Figure 7 · p. 9
8.3 BEAM SEARCH
- Beam-search performance improves with larger search budgets in every setting, but does not consistently exceed direct generation.At each step, the Judge scores continuations and the search retains the top B unfinished prefixes for further generation.Source: §8.3 · p. 9, Table 4 · p. 9
- Across the three test-time strategies, Best-of-N and revision improve responses consistently, while beam search benefits from larger budgets but remains policy- and domain-dependent.Beam search outperforms direct generation in most comparisons, but underperforms for Llama-3.1-8B on RaR-Science, particularly at smaller budgets.Source: §8.1 · p. 8, §8.3 · p. 9, §8.2 · p. 9
9 CONCLUSION
Protocol choices affect intrinsic judgment quality and downstream utility, but those effects do not always align. The conclusion calls for evaluating Judges on both how well they judge and how effectively they improve the systems they guide.
9 CONCLUSION
- Judgment quality and downstream utility do not always align, so LLM Judges should be evaluated on both judgment quality and their effectiveness in improving guided systems.Ratings can improve reward resolution and often training outcomes without consistently improving judgment accuracy or critique–verdict consistency.Source: §9 · p. 9
A IMPLEMENTATION DETAILS
This section documents the implementation across four rubric-based open-ended datasets, including Judge protocol variants and metric calculations. It also defines reward-signal statistics from logged sequence-level rewards, including response-level tie rate and group-level uniformity rate.
A.1 DATASET DETAIL
- Experiments use RaR-Medicine and RaR-Science for training, with held-out test sets and HealthBench and ResearchQA for evaluation.RaR-Medicine and RaR-Science have 17,011 and 16,365 training samples and 500 test samples each; HealthBench and ResearchQA are not used for training.Source: §A.1 · p. 12
A.2.1 T/F / CRITIQUE+VERDICT
- The default protocol evaluates each rubric item independently, generating a critique before a binary verdict.The verdict reflects whether the criterion is met, including undesirable criteria; for negative-point criteria, good responses should be classified false.Source: §A.2.1 · pp. 12, 14
A.2.2 0–10 RATING
- The 0–10 rating protocol retains single-criterion input but replaces binary verdicts with graded credit for criterion satisfaction.Scores range from 0 for no match to 10 for full satisfaction, with intermediate scores reflecting partial satisfaction; explanations should identify missing or partially met requirements below 10.Source: §A.2.2 · p. 14
A.2.3 VERDICT ONLY / A.2.4 VERDICT+CRITIQUE
- Verdict-only removes reasoning while retaining an empty explanation placeholder, whereas verdict-plus-critique generates the binary verdict before its explanation.The latter changes field and generation order while leaving inputs and criterion interpretation unchanged.Source: §A.2.3 · p. 14, §A.2.4 · p. 14
A.2.5 BATCHED T/F
- Batched T/F scores a list of rubric items in one request and returns a numbered result with an explanation and binary criterion judgment for each item.The instructions preserve per-item interpretation, including treating all required criteria as necessary and judging undesirable behaviors by whether they occur.Source: §A.2.5 · p. 15
A.3 CRITIQUE–VERDICT CONSISTENCY AUDITOR
- Separate verdict and rating auditors assess whether each explanation logically supports its corresponding verdict or score.The rating auditor accepts reasonable variation for partial-satisfaction scores and treats rubric criteria as written, including negative or pitfall criteria.Source: §A.3 · p. 16
A.4 DECISION-RULE PROMPT MODIFICATIONS
- Decision-rule robustness conditions add one evidence-threshold instruction to the unchanged neutral training-time template.Conservative variants require directly observable evidence, while permissive variants also accept coherent support from the response as a whole.Source: §A.4 · pp. 16–17
B METRIC DEFINITIONS
- The metric-definitions section provides calculations for metrics introduced in Section 4.Source: B METRIC DEFINITIONS · p. 17
B.1 REWARD-SIGNAL STATISTICS
- The response-level reward tie rate is introduced for logged sequence-level rewards organized by response group.Source: §B.1 · p. 17
- The group-level reward uniformity rate is introduced as a reward-signal statistic.Source: §B.1 · p. 17
B.2 STATIC JUDGMENT METRICS
This appendix defines the static judgment metrics and reports complementary reward-resolution and protocol results, with complete figures covering the remaining protocol and test-time comparisons.
B.2 STATIC JUDGMENT METRICS
- The metrics compare source judgments with a consensus reference, assess accuracy and critique–verdict consistency, and measure repeated-call agreement and decision-rule robustness.T/F uses majority vote; Rating uses the mean score of GPT-OSS-120B, GPT-4.1, and DeepSeek-V3.2, with distance-aware soft metrics.Source: §B.2 · pp. 17–18
- Rubric weights orient source and reference values so the positive class consistently represents a reward-beneficial outcome before confusion masses are accumulated.The metrics then derive Accuracy, TPR, and TNR from those confusion masses.Source: §B.2 · pp. 17–18
- Rating accuracy gives partial credit for score proximity: a one-point difference contributes 0.9 agreement, while scores 0 and 10 contribute zero.The reported Accuracy, TPR, and TNR are ordinary binary metrics for T/F and distance-aware soft metrics for Rating.Source: §B.2 · p. 18
C.1 REWARD RESOLUTION ACROSS CRITIQUE USAGE PROTOCOLS
- No critique protocol consistently dominates reward-resolution statistics across the evaluated settings.Table 5 reports statistics over training steps 1–80 for the default-seed trajectory.Source: §C.1 · p. 18, Table 5 · p. 19
C.2.1 REWARD RESOLUTION FIGURE OF RATING
- Figure 8 presents reward-signal statistics under T/F and Rating; lower values indicate fewer tied response pairs and fewer response groups with uniform rewards.Source: Figure 8 · p. 19
C.2.2 REWARD RESOLUTION TABLE OF BATCHING
- Batched Evaluation has higher mean performance in only six of 16 matched comparisons, while increasing response-level reward ties in seven of eight settings and group-level uniformity in all eight.The reported reward-resolution changes indicate less response-level discrimination and fewer groups with non-zero relative advantages for GRPO.Source: Table 6 · p. 19
D.1 COMPLETE STATIC JUDGMENT RESULTS
- Figures 9–11 provide complete static-judgment comparisons across two policies, two training-time Judges, and two domains, spanning verdict granularity, critique usage, and batching.Their blocks report criterion-level Accuracy, TPR, and TNR; critique–verdict consistency; repeated-call agreement; and decision-rule robustness.Source: §D.1 · p. 20, Figure 9 · p. 20, Figure 10 · p. 21
D.2 COMPLETE POSITION BIAS RESULT
- Figure 12 reports the change in True-verdict rate from independent to batched evaluation across normalized rubric positions; positive values indicate more True verdicts under batching.The supplied passage identifies the comparison but does not report its plotted outcomes.Source: Figure 12 · p. 22
E SCALING TO LARGER POLICIES
At the larger policy scale, Rating and Verdict+Critique generally retain advantages, while batching remains configuration-dependent. The appendix also specifies the common test-time protocol and three Judge-guided inference procedures.
E SCALING TO LARGER POLICIES
- The larger-policy comparison repeats the three training-time protocol comparisons with Qwen2.5-7B-Instruct to test whether the main patterns extend to a larger policy.The tables report highest validation scores across checkpoints for matched settings.Source: E SCALING TO LARGER POLICIES · p. 23, Table 7 · p. 23, Table 8 · p. 23
- Rating performs better in three of four larger-policy settings, while Verdict+Critique beats Critique+Verdict in three of four; batching splits evenly across matched settings.The authors characterize Rating and Verdict+Critique as generally preferable, but find no consistent downstream advantage for batching.Source: Table 7 · p. 23, Table 8 · p. 23, Table 9 · p. 23
F.1 COMMON PROTOCOL
- Test-time methods use fixed 200-question subsets from RaR-Science and RaR-Medicine, frozen policies, and either GPT-OSS-20B or Qwen3-30B-A3B-Instruct as Judge.Responses are sampled at temperature 0.7; Judges and the test grader use temperature 0.Source: §F.1 · p. 24
F.2 BEST-OF-N SELECTION
- Best-of-N selects the highest-scored of N complete responses, with both Judges independently scoring the same candidates against the full rubric.Each Judge assigns one overall integer score from 0 to 100; ties go to the response generated first.Source: §F.2 · p. 24
F.3 JUDGE-GUIDED REVISION
- Judge-guided revision diagnoses one to three improvement aspects, obtains rubric-blind concrete feedback, and uses that feedback for one final policy revision.The revision is not followed by additional generations or comparisons.Source: §F.3 · p. 25
F.3.1 RUBRIC-AWARE REVISION DIAGNOSIS
- Rubric-aware revision diagnosis selects one to three allowed hint codes, which guide a separate feedback call that does not receive the rubric.The listed hint codes are provided in Table 10.Source: §F.3 · p. 25, Table 10 · p. 26
F.3.2 RUBRIC-BLIND FEEDBACK GENERATION
- Rubric-blind feedback uses the visible conversation, current answer, and generic guidance to identify actionable factual, logical, mathematical, completeness, or clarity problems.The feedback prompt says not to speculate about hidden grading requirements.Source: §F.3.2 · p. 27
F.4 BEAM SEARCH
- Beam search expands partial answers, has the Judge score each resulting trajectory, and retains the B highest-scoring unfinished answers for further generation.The policy generates continuations but never receives the rubric or Judge scores.Source: §F.4 · p. 27