PaperlayerView exact revision on arXiv

Source-linked AI summary

Judging the Judges: Confidence and Disagreement in Rubrics as Rewards

Michael Tian-Yue Liu, Anne Harrington, Trevor Darrell, Yutong Bai

arXiv:2610.21179v1cs.LGcs.AIcs.CL

TL;DR

RaR uses LLM judges for non-verifiable domains, but prior work largely treated judge identity as fixed despite judges producing different scores. RaRE combines confidence-weighted ratings with filtering of high-disagreement rollouts, and outperforms every single-judge RaR variant on HealthBench and GPQA Diamond.

  • Problem

    RaR uses LLM judges for non-verifiable domains, while prior work largely treated judge identity as fixed despite different judges producing different scores.

    Source: §1 · p. 1, §5 · p. 8

  • Method

    RaRE combines a confidence-weighted soft-Likert reward using judge logprobs with a mask that excludes high-disagreement rollouts from the GRPO update.

    Source: §1 · pp. 1–2

  • Results

    RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, while naive averaging does not outperform the best single judge.

    Source: §5 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    Judge identity carries substantial signal in RaR, and RaRE’s gains are attributed to combining complementary judge strengths rather than averaging scores.

    Source: §5 · p. 8, §6 · p. 9

    Experiments use one base policy, Qwen3-4B-Base, and two domains, so findings may not transfer to other policy families, scales, or domains.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Rubrics as Rewards (RaR) extends reinforcement learning to non-verifiable domains by using an LLM judge to score responses against a structured rubric. Prior work has focused on improving the rubric or strengthening the single judge, with comparatively little attention to how different single or multiple judges might be used during RaR training. We first show that judge identity alone produces a substantial spread in scores when training Qwen3-4B-Base on RaR-Medicine and RaR-Science across five frontier and open-weight judges. Building on this, we propose RaRE (Rubrics as Rewards with Ensembles), a novel judge ensemble strategy for RaR with two mechanisms: a soft-Likert reward that uses judge logprobs to weight more confident judges more heavily, and a drop-on-disagreement mask that excludes high-disagreement rollouts from the GRPO update. We find that this design is critical: while a naive average of judge scores fails to outperform the best single judge, RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA Diamond, with per-theme analysis showing that it inherits the complementary strengths of its constituent judges. Our findings establish judge ensembling as a compelling axis for improving RaR.

1 Introduction

Prior RaR research largely focused on rubrics, leaving the training judge comparatively under-explored. This paper studies judge choice and introduces RaRE, which outperforms single-judge variants on medicine and science benchmarks.

1 Introduction

  • The study trains Qwen3-4B-Base on RaR-Medicine and RaR-Science across five open- and closed-weight judges, finding that judge identity alone substantially shifts scores.The training strategy and rubrics are held consistent with Gunjal et al. (2025).Source: §1 · p. 1
  • RaRE combines confidence-weighted soft-Likert ratings with a mask that excludes high-disagreement rollouts from GRPO updates.The two mechanisms address intra-judge confidence and inter-judge disagreement.Source: §1 · pp. 1–2
  • RaRE outperforms every single-judge RaR variant on HealthBench and GPQA Diamond, with gains across all seven HealthBench themes.The authors attribute its gains to combining complementary judge strengths.Source: §1 · pp. 1–2
  • The authors argue that judge choice is a major RaR design choice and that small ensembles with appropriate aggregation can provide a stronger training signal than any single judge.Source: §1 · p. 2

2 Related Work

Prior RaR work largely emphasizes rubric design, while judge ensembles have primarily been studied for evaluation. This paper positions judge choice and ensemble use in the training loop as complementary design directions.

2 Related Work

  • Rubric-based reward work has concentrated on rubric construction, scaffolding, and updates, while training judges are generally treated as fixed.RaRE holds the rubric fixed and treats the judge as a design variable, complementing this work.Source: §2 · p. 2
  • Reliability studies report position bias, length bias, and inter-seed variance in single judges, and this work extends the judge-choice question from evaluation to optimization.Source: §2 · p. 2
  • Prior judge-ensemble studies use panels for evaluation; RaRE uses an ensemble as the reward signal inside the RL training loop.Its mechanisms use inter-judge spread as part of training rather than only as a diagnostic.Source: §2 · p. 2

3 Methods

RaRE replaces a single training judge with an ensemble, combining confidence-weighted expected ratings with a disagreement mask. The mask removes high-disagreement rollouts from GRPO’s effective update.

3 Methods

  • RaR uses a rubric of weighted criteria and an LLM judge to produce a scalar reward; RaRE replaces the single training judge with an ensemble.The method is presented alongside single-judge RaR’s GRPO-based policy updates.Source: §3 · pp. 2–3, §3.1 · p. 3

3.1.1 Naive judge ensemble

  • The naive ensemble reward is the mean of the individual judge scores, intended to average out idiosyncratic judge bias while leaving the training pipeline unchanged.Source: §3.1.1 · p. 3

3.1.2 Soft-Likert reward via judge logprobs

  • Soft-Likert uses rating-digit logprobs to recover a distribution, then combines judges’ expected ratings using confidence weights.The confidence-weighted expected-rating average is scaled by dividing by L; inverse-variance confidence is reported as strongest among compared schemes.Source: §3.1.2 · pp. 3–4
  • When logprobs are unavailable, hard-Likert assigns confidence 1 to each parsed integer rating, reducing the ensemble to a plain average.Source: §3.1.2 · p. 4

3.1.3 Drop on disagreement

  • Drop-on-disagreement replaces high-disagreement rollout rewards with the unmasked group mean, giving those rollouts zero group-normalized advantage and no gradient contribution.Soft-Likert addresses intra-judge uncertainty, while the mask addresses inter-judge uncertainty; the mechanisms can be composed.Source: §3.1.3 · p. 4

4 Results

RaRE combines confidence-weighted judge ratings with disagreement filtering, outperforming single-judge RaR on HealthBench and GPQA Diamond. The results also show substantial judge sensitivity, complementary theme strengths, and configuration-dependent gains.

4.1 Experiment Setup

  • Experiments train Qwen3-4B-Base with GRPO on RaR-Medicine and RaR-Science, evaluating English-only HealthBench and a held-out GPQA Diamond test split.Soft-Likert experiments use judges that expose token-level logprobs; GPQA hyperparameters are tuned on a disjoint validation split.Source: §4.1 · p. 5

4.2 Single-Judge RaR is Sensitive to Judge

  • Judge choice alone shifts HealthBench scores from 0.296 to 0.339 and GPQA Diamond scores from 0.345 to 0.406 under the fixed rubric.The HealthBench spread is about 4.3 percentage points, comparable to the gap between RL and the untrained base.Source: §4.2 · p. 5, Figure 3 · p. 6
  • 97% of gpt-oss-120b post-training responses hit the 4,000-character cap versus about 4% for the untrained base; length penalties and LoRA partially mitigate expansion.The reported LoRA comparison trades a small reward decrease for substantially shorter outputs and lower training compute.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 RaRE Outperforms Single-Judge RaR

  • Naive averaging does not beat single judges, while adding drop lifts Gemini+DeepSeek HealthBench to 0.347, above the best single judge.Soft-Likert without drop scores 0.330 on DeepSeek+GPT-4o-mini; composing it with drop yields 0.364.Source: §4.3 · pp. 6–7
  • 0.364 on HealthBench and 0.412 on GPQA Diamond are RaRE’s best scores, exceeding the best single judges by +2.5 pp and outperforming them, respectively.The HealthBench result uses soft-Likert plus drop on DeepSeek+GPT-4o-mini; the GPQA result adds drop to that pair’s soft-Likert ensemble.Source: Figure 3 · p. 6, §4.3 · p. 7, Figure 2
  • RaRE’s soft-Likert-plus-drop configuration wins across HealthBench themes, combining judges whose individual strengths fall on different themes.DeepSeek wins four listed themes and GPT-4o-mini wins three; Figure 4 presents theme-level mean-reward deltas against Qwen3-4B-Base.Source: §4.3 · p. 7, Figure 4 · p. 7

4.4 Configuration Ablations

  • On HealthBench 1K, τ=0.30 is the best Gemini+DeepSeek hard-Likert threshold; on GPQA validation, τ=0.40 is selected for DeepSeek+GPT-4o-mini soft-Likert.The HealthBench sweep finds τ=0.15 masks about 14% of rollouts and τ=0.35 about 2%; the GPQA threshold masks about 24%.Source: §4.4 · p. 7, Table 3 · p. 7, Table 4 · p. 8
  • 0.364 versus 0.295 on HealthBench 1K: with drop fixed, switching from hard-Likert to inverse-variance soft-Likert adds 6.9 pp.Alternative confidence aggregation schemes score 0.214 and 0.198, both below the untrained base, indicating that confidence aggregation must account for the distribution of judge confidence.Source: §4.4 · pp. 7–8, Table 5 · p. 8

5 Discussion

With the rubric fixed, judge identity substantially shifts RaR outcomes, while ensemble gains depend on how judges are aggregated. The discussion highlights disagreement filtering and complementary judge strengths as central to RaRE’s gains.

5 Discussion

  • A ∼4.3 pp HealthBench spread across five judges exceeds the gap between Qwen3-4B-Base and the lowest single-judge RaR run.The authors argue that the judge side carries signal comparable to rubric design.Source: §5 · p. 8
  • RaRE improves on both judges across HealthBench themes, suggesting gains from combining non-overlapping strengths rather than choosing the stronger judge.Naive averaging does not outperform the best single judge; the gain appears after filtering high-disagreement rollouts.Source: §5 · p. 8
  • Adding the disagreement mask to the Gemini+DeepSeek mean lifts HealthBench by +4.0 pp (0.307 to 0.347), while removing it from DeepSeek+4o-mini soft+drop costs 3.4 pp (0.364 to 0.330).Soft-Likert adds gain when composed with the mask but alone does not beat the best single judge on HealthBench.Source: §5 · p. 8
  • Online rubric updates target rubric content, whereas RaRE targets judge aggregation; the authors say these approaches should compose, but leave testing that combination to future work.A concurrent multi-judge panel instead diagnoses reward hacking at evaluation time, while RaRE uses multiple judges to produce the training signal.Source: §5 · p. 8

6 Conclusion

The conclusion presents RaRE as a two-mechanism judge ensemble for rubric-based RL. It reports better results than every single-judge variant on both benchmarks and attributes the gains to combining complementary judge strengths.

6 Conclusion

  • RaRE combines a soft-Likert confidence-aggregation reward with a drop-on-disagreement mask.The conclusion describes these as the method’s two components.Source: §6 · p. 9
  • RaRE outperforms every single-judge RaR variant on both HealthBench and GPQA-Diamond.Source: §6 · p. 9
  • The authors suggest that RaRE’s gains arise from confidence and disagreement aggregation of complementary judge strengths.Source: §6 · p. 9

Limitations

The experiments cover one base policy and two domains, and the authors identify several boundaries on the evidence. These include evaluation-judge overlap, limited seed-based certainty, discarded rollout signal, and dependence on judge disagreement.

Limitations

  • Because experiments use only Qwen3-4B-Base on HealthBench and GPQA Diamond, transfer to other policy families, scales, or domains remains uncertain.The authors leave broader validation to future work.Source: Limitations · p. 9
  • HealthBench uses GPT-4o-mini as both a training judge and evaluator, which may introduce evaluation-judge bias despite similar rankings under GLM-5.3-Flash.A larger multi-judge evaluation panel could add robustness; GPQA Diamond uses verifier-style evaluation.Source: Limitations · p. 9
  • RaRE’s GPQA Diamond improvement over single-judge RaR is not statistically significant across five training seeds.Additional seeds could tighten confidence intervals and allow a more definitive comparison.Source: Limitations · p. 9
  • The disagreement mask discards high-disagreement rollouts entirely rather than extracting partial signal from them.Further exploration of partial-signal approaches is left to future work.Source: Limitations · p. 9
  • When judges largely agree, RaRE adds little signal beyond a single judge; a more diverse judge pool could expand where ensembling helps.Source: Limitations · p. 9

Ethical Considerations

The paper cautions that ensemble-trained policies retain judge biases and should be audited separately. Its additional analyses report category-level wins, clarify how visualizations support comparison, and test confidence intervals, evaluator overlap, and filtering with a single reward judge.

Ethical Considerations

  • Because ensemble-trained policies inherit judge biases, RaRE does not eliminate biases shared across judges, and filtering may remove cases where one judge alone flags a safety issue.The authors encourage separate bias audits, especially for health or scientific deployment, and do not treat ensembling as a substitute for evaluation.Source: Ethical Considerations · p. 9

A Full per-category breakdown

  • RaRE’s dsmini configuration wins 12 of 17 HealthBench categories outright and one GPQA topic, biology.It also performs strongly on the other two GPQA topics.Source: A Full per-category breakdown · p. 12

B Alternative visualizations of the per-theme breakdown

C HealthBench confidence intervals

  • RaRE’s HealthBench confidence interval is fully above every single-judge confidence interval in the English-only subset.The reported intervals are bootstrapped 95% confidence intervals.Source: C HealthBench confidence intervals · p. 12

D Independent HealthBench judge

  • Under GLM-5.3-Flash regrading, relative rankings remain similar and RaRE ranks first; single-judge policies trained with GPT judges drop most, while RaRE changes little.The regrade uses the same 3,931 English prompts and grading prompt.Source: D Independent HealthBench judge · p. 12

E Single-judge reward with two-judge filtering

  • Using a second judge only to compute the disagreement mask yields 34.2 on HealthBench, above single-judge GPT-4o-mini RaR (33.9) but below RaRE (36.4).The result indicates that filtering with a second judge helps even when only one judge supplies the reward; the soft-Likert ensemble reward adds further gains.Source: E Single-judge reward with two-judge… · p. 12

F Training cost and latency

RaRE scores higher than single-judge GPT-4o under both evaluators at about one quarter of its API cost, while its sequential two-judge implementation increases reward latency.

F Training cost and latency

  • 2.06 s is RaRE’s median per-batch reward latency, versus 1.01 s for a single GPT-4o-mini judge.The two judges are queried sequentially in this implementation; querying them in parallel would reduce the overhead.Source: F Training cost and latency · p. 13
  • RaRE scores higher than GPT-4o under both evaluators at about one quarter of the API cost.Table 9 reports training-time judge API cost and HealthBench score.Source: F Training cost and latency · pp. 12–13, Table 9 · p. 13

G Disagreement case studies

Disagreement examples show judges applying explicit rubric items differently, with GPT-4o-mini rating more leniently than DeepSeek in the reported masked rollouts.

G Disagreement case studies

  • On a response contradicting an explicit ethics rubric item, GPT-4o-mini rated it 4.99 while DeepSeek rated it 1.96.The rubric explicitly said the child’s refusal to donate bone marrow should not be overridden, but the response concluded that overriding it could be justified.Source: G Disagreement case studies · p. 14
  • For an answer that omitted the Migrating Motor Complex despite correctly selecting Motilin, GPT-4o-mini rated it 4.00 and DeepSeek 2.63.The rubric specified both Motilin and the Migrating Motor Complex as essential items.Source: G Disagreement case studies · p. 14
  • 25.35% of rollouts were masked, and GPT-4o-mini gave the higher rating in 99.61% of masked rollouts.Across 25,600 training rollouts from four two-judge runs, the pattern suggests GPT-4o-mini applies rubric items more leniently than DeepSeek.Source: G Disagreement case studies · pp. 14–15

H GPQA answer-order evaluation

Under ten answer-order permutations, RaRE’s mean GPQA Diamond accuracy exceeded every single-judge RaR policy but was not significantly different from them. The accompanying evaluation details its protocol, HealthBench language scope, and judge prompts.

H GPQA answer-order evaluation

  • Under ten answer-order permutations, RaRE’s mean GPQA Diamond accuracy exceeded every single-judge RaR policy, but the differences were not statistically significant.Each row is a single trained checkpoint evaluated ten times; the main-text results instead average five independently trained checkpoints. Table 10 reports all 198 questions and mean ± 2 standard errors over permutations.Source: H GPQA answer-order evaluation · p. 15, Table 10 · p. 15

I English-only HealthBench evaluation

  • HealthBench evaluation uses English prompts because rubric–response language alignment may confound multilingual comparisons; localized-rubric evaluation is left to future work.All HealthBench rubrics are in English; exploratory checks found higher scores for English responses to Spanish prompts and higher average grades after translating rubrics into Spanish.Source: I English-only HealthBench evaluation · p. 15

J Judge prompts

  • The study uses rubric-conditioned Likert-rating and plain pairwise-judge prompt families, with strict JSON decoding constraining outputs to a rating or winner.Both prompt families return no rationale.Source: J Judge prompts · p. 15

Rubric Likert judge (scale 1–5)

  • The rubric Likert judge rates response quality from 1 to 5 against the prompt, response, and listed rubrics; the rendered criteria are an unweighted flat list.For gemds runs, the scale is 1–10 with generalized verbal anchors.Source: Rubric Likert judge (scale 1–5) · pp. 15–16

Pairwise (no-Likert) judge

  • The pairwise prompt asks a judge to choose which of two responses better satisfies the rubrics, returning only A or B.The no-rubric ablation removes the rubric field and wording, leaving the judge to assess accuracy, helpfulness, depth, and clarity implicitly.Source: Pairwise (no-Likert) judge · p. 16

K Soft-Likert rating scale ablation

The appendix evaluates soft-Likert scale and disagreement-threshold choices, LoRA, pairwise rewards, and retroactive reuse of dropped rollouts, alongside implementation and artifact details.

K Soft-Likert rating scale ablation

  • 0.263 for L=5 slightly exceeds 0.254 for L=7 on the HealthBench 1K subset; the wider rating scale does not improve the point estimate.The authors chose L=5 for soft-Likert, whose logprob averaging permits decimal averages despite the smaller scale.Source: K Soft-Likert rating scale ablation · p. 16, Table 11 · p. 16

L DeepSeek+Mini soft-Likert τ sweep

M LoRA vs. full-FT comparison

  • At matched compute budgets, LoRA costs ∼3 pp on the 1k subset, and its reward signals are nearly indistinguishable, suggesting adapter capacity is the binding constraint.LoRA also produces ∼2–4× shorter responses on average, useful when the policy tends to length-hack.Source: M LoRA vs. full-FT comparison · pp. 16–17, Table 13 · p. 17

N Pairwise (non-rubric) baselines

  • Pairwise mini/ds reach roughly the level of their hard-Likert RaR counterparts, while pairwise gem collapses; the rubric absorbs most calibration burden in RaR.The baseline provides the judge two candidate responses without a rubric and asks for a binary preference.Source: Table 14 · p. 17, N Pairwise (non-rubric) baselines · p. 17

O Retro-GRPO-aux

  • All tested retro variants were null or mildly negative versus their drop030 baselines; full-FT variants OOM’d before convergence on 80 GB H100s.For this judge-pair and policy combination, the authors could not extract reusable signal from dropped rollouts beyond group-mean substitution.Source: O Retro-GRPO-aux · p. 17

P Training hyperparameters

Q Model size and budget

  • The experiments train a 4-billion-parameter Qwen3-4B-Base model on one 8-H100 node; the best medicine and science methods take 8 H100-hours total.The reported total training time is one hour.Source: Q Model size and budget · p. 17

S Artifact licenses

  • The study uses HealthBench, GPQA Diamond, Qwen3-4B-Base, and RaR-Medicine/Science under their respective licenses; code is promised upon publication.Judge models were accessed through providers’ APIs under their terms of service.Source: S Artifact licenses · p. 17
Loading 2610.21179v1…