PaperlayerView exact revision on arXiv

Source-linked AI summary

Small Language Models as Judges for Rubric-Based Reinforcement Learning

Fengyu Xie, Yilun Zhao, Bingsen Chen, Arman Cohan, Chen Zhao

arXiv:2608.30005v1cs.CL

TL;DR

Rubric-based RL needs repeated, costly judgments of responses against instance-specific criteria, motivating whether smaller models can judge rubrics effectively. The paper constructs pointwise benchmarks, compares three judgment readouts, and finds Probe judges strongest across datasets, with transfer evidence beyond the calibration setting and better RL results than an 8B Generative baseline.

  • Problem

    Rubric-based RL requires expensive repeated scoring by proprietary APIs or large generative judges, creating a need to assess whether much smaller models can serve as effective rubric judges.

  • Method

    The paper constructs PointRubric and RaR-Science-Static, then compares Generative, Yes/No Logprob, and hidden-state Probe judges using Qwen3 models from 0.6B to 8B parameters.

  • Results

    Probe judges perform best across both datasets; the Qwen3-1.7B Probe trains a policy from 0.232 to 0.643 RaR-Science rubric score versus 0.594 for an 8B Generative baseline, while transfer checks extend to GPQA-Diamond and RaR-Medicine.

  • Takeaways & Limitations

    Small Probe judges provide an effective and efficient reward model for rubric-based RL, preserving criterion-level reward structure across the tested task and domain transfers.

  • Takeaways & Limitations

    The evidence is strongest on RaR-Science, GPQA-Diamond, and RaR-Medicine, and broader human evaluation and calibration may be needed for new domains and settings.

Abstract

from arXiv · show

Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7$\times$ more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.

1 Introduction

Rubric-based RL extends reward verification to responses whose quality depends on multiple instance-specific criteria, but repeated LLM judging is costly and difficult to reproduce. The paper tests smaller judges, introduces matching pointwise benchmarks, and finds Probe-based judging strongest for both evaluation and RL reward computation.

  • Motivation: Rubric-based RL evaluates content coverage, factuality, source use, and task-specific constraints through instance-specific criteria while producing scalar rewards.These criteria target long-form generation settings that simple automatic checkers do not capture well.
  • Motivation: Repeated rubric judging makes reward computation expensive, while proprietary judges hinder exact reproduction.Each RL step may score long responses against multiple criteria, and access can depend on a fixed provider model snapshot.
  • Datasets: PointRubric and RaR-Science-Static provide pointwise rubric-evaluation data with controlled responses, criterion-level labels, and weighted scoring structures.The datasets are derived from OpenRubrics and RaR-Science to match rubric-based RL reward computation.
  • Judge comparison: Probe judges substantially outperform Generative and Logprob judges across both static settings, indicating that small models encode useful evaluative signals in hidden states.The comparison uses Qwen3 models ranging from 0.6B to 8B parameters and trains only a lightweight classifier for Probes.
  • RL results: 0.232 to 0.643: the Qwen3-1.7B Probe reward improves the policy’s RaR-Science rubric score, versus 0.594 for an 8B Generative judge baseline.The baseline requires 10.7× more reward-judge time at the comparison checkpoint.

2 Rubric Judging Task and Dataset

The paper defines pointwise rubric judging as criterion-level satisfaction prediction and builds datasets that preserve rubric structure for evaluation and reward calculation. PointRubric and RaR-Science-Static provide complementary controlled settings with weighted aggregation and GPT-4o reference labels.

  • 2.1 Our Task: Pointwise Rubric Judging: Pairwise preference data identifies a relative winner but does not reveal which response aspects are correct or incorrect.Pointwise scoring can also induce pairwise preferences after aggregating criterion scores, while retaining the rubric structure.
  • 2.1 Our Task: Pointwise Rubric Judging: Pointwise rubric judging predicts whether one candidate response satisfies one rubric criterion, preserving criterion-level structure for reward computation.Binary or soft criterion outputs are aggregated using rubric weights into scalar scores.
  • 2.2 Dataset Design Overview: Pointwise rubric evaluation requires explicit criteria, multiple responses per prompt and rubric, per-criterion labels, and weighted aggregation.Existing resources commonly lack criterion-level labels, controlled response banks, or the weighted structure used in rubric-based RL.
  • 2.3 PointRubric Construction: PointRubric converts OpenRubrics into a controlled pointwise benchmark by adding candidate responses and per-criterion satisfaction labels.Its rubric sources provide general-domain prompts and detailed natural-language criteria.
  • 2.3 PointRubric Construction: PointRubric standardizes each sampled rubric into six criteria: two hard constraints and four soft quality criteria.A manual audit found that 47 of 50 sampled transformations preserved the original rubric’s core intent.
  • 2.3 PointRubric Construction: 1,042 questions, 4,168 responses, and 25,008 response–criterion labels comprise the validated PointRubric benchmark.Question-level splits contain 417 training, 104 development, and 521 held-out questions, keeping all responses to a prompt together.
  • 2.4 RaR-Science-Static: RaR-Science-Static applies the same pointwise judge methods to science rubrics with variable criteria and absolute criterion weights, separately from RL experiments.It complements PointRubric by matching the downstream science-domain rubric format.
  • 2.5 Reference Labels and Human Audit: GPT-4o supplies operational reference labels rather than human ground truth, and a human audit found 90.9% weighted criterion agreement.The audit covered 600 decisions and 50 paired comparisons; disagreements concentrated in softer qualitative criteria.

3 Rubric-Judge Methods

The paper compares three criterion-level readouts—Generative, Logprob, and Probe—using the same prompt, response, criterion, and weighted aggregation pipeline. Probes expose hidden-state signals through a lightweight classifier, while SFT tests whether verdict-prediction training changes judge quality.

  • 3 Rubric-Judge Methods: All judge methods take the same item-level input (q, y, cj) and aggregate criterion outputs with the same rubric weights.The methods differ in how they read out criterion satisfaction from the backbone.
  • 3 Rubric-Judge Methods: The study compares Generative verdicts, Yes/No Logprob scoring, and hidden-state Probes, with SFT changing the backbone state.Generative, Logprob, and Probe define the criterion-level readout in the method matrix.
  • 3.1 Scoring Methods: Generative judges produce binary satisfaction verdicts that are parsed and then aggregated with rubric weights.Outputs that cannot be parsed into the required binary format count as invalid predictions.
  • 3.1 Scoring Methods: Logprob judges compute a Yes/No log-probability margin instead of generating a verdict.The margin is thresholded at 0 for binary metrics, while σ(∆j) supplies the soft satisfaction score for aggregation.
  • 3.1 Scoring Methods: Probe judges freeze the language model, extract a final hidden state at a selected layer, and train only a lightweight linear classifier on GPT-4o labels.The classifier is trained with binary cross-entropy to predict criterion satisfaction.
  • 3.2 Post-training with Supervised Fine-tuning: SFT trains a backbone on prompts, candidate responses, complete rubrics, and GPT-4o Yes/No verdict vectors before reevaluating all three readouts.This tests whether supervised rubric-verdict prediction makes the backbone a better rubric judge.

4 Static Rubric-Judge Evaluation

The evaluation compares Generative, Logprob, and Probe readouts across controlled rubric benchmarks and model scales. Probes achieve the strongest RaR-Science-Static agreement, remain effective under response shifts, and are selected for RL based on downstream policy improvement.

  • 4 Static Rubric-Judge Evaluation: The study evaluates Generative verdicts, Yes/No Logprob scoring, and hidden-state Probes across Qwen3 backbones from 0.6B to 8B parameters.All methods use the same held-out splits, with GPT-4o criterion labels as the operational reference.
  • 4.1 Experiment Setup: PointRubric uses weighted criterion accuracy, while RaR-Science-Static uses criterion-level macro-F1 as its primary metric.The metrics reflect each benchmark’s criterion weighting, rubric length, label balance, and downstream scoring setting.
  • 4.2 Static Judge Results: 0.902 weighted criterion accuracy is achieved by the 1.7B SFT Generative judge on PointRubric, compared with 0.370 for the 0.6B base Generative judge.Base Generative performance rises to 0.888 at 8B, showing scale improvement in the controlled benchmark.
  • 4.2 Static Judge Results: 0.835 macro-F1 is reached by the 1.7B Probe on RaR-Science-Static, versus 0.443 for Generative and 0.449 for Logprob using the same backbone.Among 4,518 held-out decisions, 33.0% are correct only under Probe, compared with 7.8% correct only under Generative.
  • 4.2 Static Judge Results: Probe is the strongest RaR-Science-Static readout across model sizes, with 0.835 macro-F1 at 1.7B and 0.864 at 8B.Under response-generator shift, the unchanged 1.7B Probe reaches 0.741 macro-F1, versus 0.394 for Generative and 0.424 for Logprob.
  • 4.2 Static Judge Results: Qwen3-1.7B Probe reaches pairwise accuracies of 0.924 on PointRubric and 0.888 on RaR-Science-Static when compared with a pairwise rubric reward-model baseline.The baseline’s best accuracies are 0.322 and 0.487 when parse failures count as incorrect, or 0.575 and 0.670 on parseable outputs.
  • 4.3 Selecting the Probe Configuration for RL: 0.805 macro-F1 is obtained with 100 training questions and 0.834 with 1,000 for the linear last-token Qwen3-1.7B Probe.MLP heads and last-4-layer averaging provide only modest gains, while mean pooling is substantially worse.

5 Probe-Based Rubric Rewards for RL

The study tests Qwen3-1.7B Probe judges as RL reward models under matched conditions, finding stronger policy scores than an 8B Generative judge and transfer beyond the training setting. It also examines why static judge accuracy does not perfectly predict downstream reward utility and quantifies the efficiency-quality tradeoff.

  • 5.1 Matched RL Protocol and Results: The matched protocol fixes the actor, data, GRPO configuration, aggregation, checkpoint, and evaluator, varying only the reward judge.All controlled runs use the same Qwen3-4B-Base actor and RaR-Science training set.
  • 5.1 Matched RL Protocol and Results: 0.643 RaR-Science rubric score is achieved by the Qwen3-1.7B Probe reward, versus 0.594 for the Qwen3-8B Generative reward.Both policies start from the same base actor and use the matched training and evaluation protocol.
  • 5.1 Matched RL Protocol and Results: Static judge accuracy does not exactly predict downstream reward utility: the 1.7B Probe produces the strongest policy despite larger Probes having slightly higher fixed-bank macro-F1.On-policy diagnostics suggest larger Probes may provide less within-group reward variation because of more saturated rewards.
  • 5.1 Matched RL Protocol and Results: The blind audit prefers the probe-reward policy over the base actor in 72 of 100 examples, while GPT-4o favors it in 80.Human and GPT-4o preferences agree on 69 of 72 non-tied cases, or 95.8%.
  • 5.2 Transfer Results: 0.388 GPQA-Diamond accuracy is reached by the probe-reward policy, up from 0.335 across 198 questions and four answer-order runs.This result suggests policy improvement beyond the RaR-Science rubric-scoring format.
  • 5.2 Transfer Results: 0.718 macro-F1 is obtained when a RaR-Science-trained Qwen3-1.7B Probe is evaluated on RaR-Medicine without Medicine-label fitting.Performance is below the Medicine-trained in-domain probe but retains meaningful agreement under a different rubric distribution.
  • 5.3 Efficiency-Quality Results: The efficiency comparison fixes the RL pipeline while contrasting Qwen3-1.7B Probe with Qwen3-8B Generative reward judges.The Probe is trained once on 500 GPT-4o-labeled responses and can be reused in the same rubric setting without further GPT-4o calls or recalibration.
  • 5.3 Efficiency-Quality Results: 2.33 judge-hours versus 24.98 judge-hours yields the higher Probe policy score, 0.643 versus 0.594.Parallelized reward calls make wall-clock reduction smaller, but the Probe still requires substantially less reward-judge computation.

6 Related Work

The paper positions Probe judges as a smaller alternative to large or proprietary evaluators for rubric-based RL, replacing repeated large-judge calls with a smaller criterion-level evaluator.

  • Rubric-based RL: Rubric-based RL extends verifiable-reward methods to open-ended tasks requiring broader feedback than rule-based correctness checks.The setting uses instance-specific science rubrics and downstream scalar rewards.
  • Efficiency-quality comparison: The Qwen3-1.7B Probe judge is reported to produce a higher-scoring policy than the generative baseline while using far less judge time.Table 8 frames the comparison as both a policy-quality and reward-computation tradeoff.
  • LLM-as-a-judge and small evaluators: The paper replaces repeated calls to a large judge with a smaller criterion-level evaluator.This is the central efficiency-oriented distinction from the related evaluator approaches discussed.
  • LLM-as-a-judge and small evaluators: LLM-as-a-judge methods score, compare, or critique open-ended responses, while trained open judges fine-tune models for evaluation.Related rubric-oriented work includes calibrated multidimensional evaluation.

7 Conclusion

The paper introduces datasets and Probe readouts for studying small rubric judges, then evaluates their reward use and transfer across tasks and domains. Its conclusion is that small Probe judges provide an effective and efficient rubric-based RL reward model within the tested settings.

  • 7 Conclusion: PointRubric and RaR-Science-Static provide evaluation settings for testing small judges on criterion-level rubric satisfaction.The datasets are derived from OpenRubrics and RaR-Science, respectively.
  • 7 Conclusion: Across both datasets, Probe judges perform best and show useful rubric-satisfaction signals in 1.7B-scale hidden states.The conclusion contrasts Probe readouts with the alternative judging methods evaluated in the paper.
  • 7 Conclusion: The Qwen3-1.7B Probe trains a policy that outperforms the 8B Generative reward baseline while using far less judge time.This conclusion summarizes the controlled RL and efficiency comparison.
  • 7 Conclusion: Transfer experiments find improved GPQA-Diamond policy accuracy and retained criterion-level agreement on RaR-Medicine.These results cover both policy transfer and judge transfer.

Limitations

The evidence is strongest in selected science and transfer settings, with human evaluation, RL configuration, and efficiency results subject to explicit scope boundaries.

  • The supervised target is GPT-4o criterion scoring rather than direct human ground truth, despite supporting audits.Broader human evaluation would be needed to characterize agreement across tasks and domains.
  • Evidence is strongest on RaR-Science, with additional checks on GPQA-Diamond and RaR-Medicine rather than domain-invariant validation.New domains, especially dialogue, long-form instruction following, or safety-critical expert settings, may require additional calibration and validation.
  • The RL experiments cover one policy family and one rubric-RL setup, so audits against reward overoptimization remain setting-specific.The actor, reward aggregation, and training configuration were controlled.
  • 10.7× more reward-judge time is an empirical implementation-and-hardware measurement, not a universal serving constant.Parallelized reward calls and other RL costs make judge-time advantages larger than wall-clock advantages.

Ethical considerations

The work reuses public datasets and conducts limited internal audits, while acknowledging that derived artifacts may retain unsafe or sensitive content and are not deployment guarantees.

  • The datasets reuse OpenRubrics and RaR-Science, with derived artifacts required to retain source attribution and follow original usage terms.PointRubric adds rubric-derived structure, while RaR-Science-Static adds fixed responses and reference labels.
  • No private user data, demographic attributes, or personal identifiers were intentionally collected or retained.Sampled prompts, answers, and responses were manually inspected for obvious personally identifying information and unsafe or offensive content.
  • Residual unsafe, biased, sensitive, or offensive content may remain inherited from source artifacts or produced by models.The derived datasets are therefore treated as research-only evaluator resources, not guarantees of judge safety for deployment.
  • Two small audits used research-team annotators evaluating rubric compliance or pairwise response preferences and reported only aggregate statistics.No external recruitment or payment was involved, and annotators were not asked to disclose personal information.

A PointRubric Construction and Rubric Scoring

The paper constructs pointwise rubric benchmarks and formalizes weighted criterion scoring, then validates Probe judges and the benchmark’s reference labels across controlled evaluation settings.

  • A.1 Pointwise and Pairwise Rubric Scoring: Weighted criterion judgments are aggregated into scalar rubric rewards, preserving rubric structure for evaluation and reinforcement learning.Criteria have nonnegative weights, and probabilistic judges output criterion-satisfaction probabilities.
  • A.1 Pointwise and Pairwise Rubric Scoring: Pointwise judging evaluates one response against one rubric criterion, preserving criterion-level satisfaction information that pairwise preferences collapse.Pairwise preferences can be induced from pointwise scores, but cannot recover which criteria were satisfied or violated.
  • A.2 PointRubric Construction Details: PointRubric construction standardizes rubrics into two hard and four soft criteria, retaining questions whose responses span full, low, and intermediate scores.The benchmark uses GPT-4o for rubric refinement and reference grading.
  • A.3 Rubric-Transformation Audit: 47 of 50 audited transformations preserved core rubric intent, all avoided unsupported additions, and 49 produced reasonable hard/soft decompositions.The three intent failures involved omitted requirements during compression rather than newly introduced requirements.
  • A.4 PointRubric Response Roles: The benchmark’s four response roles provide fixed full-score and low-score anchors plus partial and unguided responses with broader intermediate variation.These roles are not treated as ordinal quality bins.
  • B.2 Probe Training and Calibration: Probe judges substantially outperform Generative and Logprob judges across the two static settings, using frozen backbones with lightweight classifiers trained on GPT-4o labels.The evaluation protocol uses a fixed question-level split and weighted criterion accuracy as the primary PointRubric metric.

B.6 Static Readout Error Analysis

The analysis compares criterion-level readouts and finds that Probe judgments generally reduce Generative false positives, while still making distinct errors on some individual decisions.

  • Readout error patterns: Probe judges remove many Generative false positives, but their remaining errors are more balanced between false positives and false negatives.The paired analysis covers 4,518 held-out criterion decisions.
  • Readout error patterns: 7.8% of decisions favor Generative over Probe, showing that Probe does not uniformly dominate at the individual-decision level.
  • Qualitative error cases: A Generative false positive accepts a response that names nuclear fusion but omits the required mass-to-energy explanation.
  • Qualitative error cases: A Probe false negative rejects an explicitly exponential wavefunction that Generative and GPT-4o mark as satisfying the criterion.
  • Metric and parsing considerations: Generative judges require verdict decoding and parsing, whereas Logprob and Probe methods produce criterion scores without generative verdict parsing.Generative sample-level parse success is the fraction of responses yielding a complete criterion verdict vector; criterion-level parse success concerns individual verdicts.
  • Metric and parsing considerations: The evaluation reports criterion accuracy, weighted criterion accuracy, macro-F1, scalar-score error, score correlation, and pairwise ordering.Weighted accuracy uses rubric weights, while macro-F1 treats satisfied and unsatisfied criteria as separate classes.
  • Probe layer selection: The selected Probe layer achieves 0.833 held-out macro-F1, close to 0.834–0.835 for other Probe fits, indicating a broad middle-to-late-layer plateau.

C Rubric RL Details

The controlled RL appendix isolates the reward judge while documenting reward construction, efficiency measurement, on-policy dynamics, human validation, and transfer evaluations.

  • C.1 RL Reward Configuration: The controlled RL comparison fixes the actor, data, rollout, optimizer, reward aggregation, checkpoint, and evaluator, varying only the reward judge.
  • C.1 RL Reward Configuration: The 8B Generative baseline decodes per-criterion Yes/No verdicts, while the Probe reward uses a frozen model and calibrated linear classifier for criterion-satisfaction probabilities.
  • C.2 RL Score Confidence Intervals: Final RL score intervals resample the 100 reserved RaR-Science evaluation prompts with replacement.
  • C.3 On-Policy Reward Dynamics and Qualitative Cases: The 8B Probe is nearly constant within sampled rollout groups, whereas the 1.7B Probe and 8B Generative judge preserve more within-group reward variation.A group is fully saturated when every candidate receives a reward above 0.95.
  • C.3 On-Policy Reward Dynamics and Qualitative Cases: The saturated 8B-Probe rewards provide little within-group ranking signal, while the 1.7B Probe produces the strongest externally evaluated policy in the controlled comparison.This motivates selecting reward judges by downstream policy quality rather than fixed-bank accuracy alone.
  • C.4 Efficiency Measurement: 31.1 seconds of probe judge time compares with 492.0 seconds for the generated reward across the same 50 responses and 385 criteria.The probe uses hidden-state classification, while the Generative reward decodes and parses Yes/No verdicts.
  • C.4 Efficiency Measurement: Training elapsed time falls from 11h 11m 56s to 8h 51m 47s, a 2h 20m 09s reduction; cumulative judge time remains the primary efficiency metric.Elapsed time includes other shared pipeline components and concurrent reward calls.
  • C.5 Human Audit and Reference-Judge Validation: Human auditors favor the probe-reward policy on 72 examples versus 6 for the base actor, with 13 ties and 9 unsure labels.Excluding human Tie/Unsure labels and GPT-4o exact ties, human and GPT-4o preferences agree on 69 of 72 examples.

F Dataset Examples

The examples show how pointwise rubrics separate partially correct responses from fully satisfied criteria in software and science settings.

  • PointRubric example: The PointRubric example evaluates an SBOM-based installation comparison response against multiple criteria.
  • PointRubric example: 9 / 10 is the GPT-4o score for the PointRubric response, with verdicts Yes, Yes, Yes, No, Yes, Yes.
  • RaR-Science example: The RaR-Science example asks for the mean time between collisions from a gas’s velocity and mean free path.
  • RaR-Science example: The science response applies τ = λ/ū correctly but omits conversion from most probable velocity to average velocity before using the collision-time formula.
  • RaR-Science example: 0.773 is the GPT-4o score for the RaR-Science response, whose verdicts contain one No among otherwise Yes labels.
Loading 2608.30005v1…