PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubrics from the Task, Verdicts from Evidence: Pointwise Rubric Rewards with Tool-Augmented Judges

Tianci Liu, Haoyu Wang

arXiv:2610.18766v1cs.CL

TL;DR

Rubric-based reward models face response-induced criteria, difficulty checking mechanical constraints by reading alone, and pairwise judgments that complicate use in policy optimization. ReVeR derives typed rubrics from instructions and uses tool-augmented pointwise judging, trained through label-blind distillation. It outperforms pairwise rubric RMs in average preference accuracy and downstream policy performance, while the paper finds these evaluation dimensions do not necessarily align.

  • Problem

    Response-induced rubrics may limit generalization to unseen responses, reading-only judges can struggle with mechanical constraints, and pairwise preferences complicate direct use as policy rewards.

    Source: §1 · p. 2

  • Method

    ReVeR reasons from the instruction to create hard rules and graded principles, judges responses independently with tool-assisted mechanical checks, and trains both components through label-blind distillation.

    Source: §1 · p. 2, §3.2 · p. 4, §3.4 · p. 6

  • Results

    ReVeR has the highest average preference accuracy among single-judgment reward models and improves on pairwise rubric rewards in expert-rubric agreement, best-of-N selection, and policy optimization.

    Source: §5 · p. 12

  • Takeaways & Limitations

    Rubric fidelity, preference accuracy, and downstream utility are distinct reward properties that need to be measured separately.

    Source: §1 · p. 2, §5 · p. 12

Abstract

from arXiv · show

Rubric-based reward models (RMs) evaluate responses against explicit criteria. However, existing methods often derive criteria from observed response candidates, introducing response-induced bias that may limit rubric generalizability. Their judges also struggle with mechanical constraints when relying on reading alone, and commonly used pairwise judges provide only relative preferences, complicating their direct use for policy optimization. We propose ReVeR, a pointwise rubric RM that derives hard rules and graded principles by reasoning about the instruction alone and evaluates responses independently through tool-augmented judging. Both components are trained through label-blind distillation, in which preference labels only filter a teacher's outputs, yielding ReVeR-Traces, a large-scale dataset of rubric-reasoning and judging trajectories. Experiments demonstrate that ReVeR outperforms pairwise rubric RMs in average preference accuracy, most clearly on instruction following, as well as in test-time scaling and downstream policy optimization. We further find that generator RL improves preference accuracy while reducing rubric fidelity, and that higher preference accuracy does not necessarily translate into better downstream policies. These findings highlight that rubric fidelity, preference accuracy, and downstream utility are distinct properties, shedding light on future RM research.

1. Introduction

Existing rubric reward models can inherit criteria shaped by observed responses, struggle to verify mechanical constraints by reading alone, and provide pairwise rather than per-response rewards. ReVeR derives typed rubrics from instructions, judges responses independently with tools, and is reported to improve preference accuracy and downstream policy performance over pairwise rubric RMs.

1. Introduction

  • Response-induced rubrics may inherit candidate-specific bias, limiting generalization to unseen responses, while reading-only judges can struggle to verify mechanical constraints and pairwise verdicts complicate policy optimization.These limitations motivate criteria grounded in the instruction and independent response scores.Source: §1 · p. 2
  • ReVeR derives hard rules and graded principles from the instruction alone, then judges each response independently with a sandboxed Python interpreter for mechanical checks and reasoning for semantic criteria.The final reward prioritizes hard-rule satisfaction and uses principle scores to distinguish responses that satisfy the same number of hard rules.Source: §1 · p. 2
  • Label-blind distillation generates rubric reasoning from instructions and judges preferred and rejected responses in separate episodes; preference labels only retain examples whose scores agree with them.The process produces ReVeR-Traces with rubric-reasoning traces and tool-augmented judging trajectories; the generator is trained with SFT followed by RL.Source: §1 · p. 2
  • ReVeR is a 4B pointwise rubric RM, and ReVeR-Traces provides rubric-reasoning and judging trajectories as a training resource.The paper describes the dataset as large and diverse and releases it with model checkpoints.Source: §1 · pp. 2–3
  • ReVeR achieves higher average preference accuracy than pairwise rubric RMs, leads on every instruction-following benchmark, and delivers stronger average downstream performance under DPO and GRPO.The reported policy gains are clearest on instruction following; the paper also finds that rubric fidelity, preference accuracy, and downstream utility do not necessarily align.Source: §1 · pp. 2–3

2. Related Work

Related work covers rubric and checklist rewards, reasoning and tool-augmented judges, and open evaluator-training data. ReVeR differs by deriving criteria from instructions, producing per-response verdicts, and recording label-blind reasoning and tool trajectories.

Rubric and checklist rewards

  • Rubrics and checklists decompose evaluation into explicit criteria and serve as RL rewards, supervision, or components jointly optimized with judges.Prior criteria may be written from instructions or induced from preferred/rejected answers and sampled-candidate failure modes; ReVeR instead derives criteria by reasoning from the instruction and returns one verdict per criterion per response.Source: §2 · p. 3

Reasoning and tool-augmented judges

  • Prior judges use reasoning or retrieved, agent-gathered, or executed-code evidence, while ReVeR lets the judge choose per criterion whether to execute a check and trains by distillation alone.Examples include RewardAgent's code-based verifier and TIR-Judge's tool-integrated reinforcement learning.Source: §2 · p. 3

Open data for training evaluators

  • Open evaluator resources differ in whether they record contrastive rubrics, per-criterion labels, sampled-response checklists, reference-based rubrics, feedback, or distilled reasoning.ReVeR-Traces records reasoning behind rubrics and tool-augmented trajectories behind verdicts, generated label-blind.Source: §2 · pp. 3–4

3. Method

ReVeR generates instruction-specific rubrics and judges responses independently, combining rubric reasoning, tool-checked verdicts, and fine-grained aggregation. It trains its generator and judge from filtered teacher traces and also optimizes the generator against preference judgments.

3.1. Problem Formulation

  • A rubric is generated from the instruction, while each response is judged independently under that shared rubric and its verdicts are aggregated into a reward.The score is invariant to other responses under consideration and their presentation order.Source: §3.1 · p. 4

3.2. Task-Derived Rubric Generation

  • Rubrics reason from the instruction to identify deliverables, requirements, inferred expectations, and failure modes, then type criteria as mandatory hard rules or graded principles.A hard rule is non-negotiable; a principle measures quality by degree.Source: §3.2 · p. 4

3.3. Pointwise Tool-Augmented Judging

  • Mechanical criteria are checked against executed Python outputs, while semantic criteria rely on reasoning; hard rules score 0 or 1 and principles receive integer scores from 0 to 100.Aggregation is lexicographic: hard-rule counts take priority, and principles order responses tied on that count.Source: §3.3 · pp. 5–6

3.5. ReVeR-Traces

  • 30,310 of 41,388 preference triples pass the acceptance test, yielding 30,310 rubric-reasoning traces and 60,620 judging trajectories across five preference datasets.Each retained pair contributes two judging trajectories.Source: Figure 2 · p. 6, §3.5 · p. 7

3.4. Label-Blind Distillation

  • The teacher generates rubrics from instructions and judges preferred and rejected responses in separate episodes; labels only filter agreeing, well-formed samples without changing their content.The retained sequences are generated only from information available to the corresponding students at inference.Source: §3.4 · p. 6

3.6. Generator Reinforcement Learning

  • Generator RL optimizes preference discrimination with the judge fixed; preference enters as output reward, preserving instruction-conditioned rubrics and pointwise scoring but targeting discrimination rather than task fidelity.The RL-trained and distilled-generator configurations share one judge.Source: §3.6 · p. 7

4. Experiments

The experiments compare ReVeR with other reward models on preference accuracy, response selection, policy optimization, and agreement with expert rubrics. Across these evaluations, ReVeR performs strongly, while generator RL improves some measures and reduces rubric agreement.

4.1. Reward-Model Benchmarks

  • Training raises the untrained Qwen3.5-4B average from 57.0 to 68.1 with ReVeR (SFT); five judgments raise the SFT and G-RL averages to 73.5 and 73.8.With the judge shared, ReVeR (G-RL) averages 1.0 point above SFT, while five judgments per response put both above the teacher’s single-judgment average of 71.7.Source: §4.1 · p. 8
  • ReVeR (G-RL) and ReVeR (SFT) achieve the two highest single-judgment averages, 69.1 and 68.1, ahead of Rubric-ARM-8B (66.3) and 32B reasoning models.ReVeR leads every instruction-following column and RubricBench Chat; 32B reasoning models lead on RewardBench, RM-Bench, and JudgeBench.Source: §4.1 · p. 8, Table 2 · p. 8

4.2. Downstream Evaluation

  • At N = 16, ReVeR reaches 85.1 on IFEval and 60.3 on MulDimIF, 3.7 and 5.1 points above the strongest baseline.These results represent 67% and 55% of the respective checker-oracle gains; with four candidates, both configurations already exceed every baseline at sixteen.Source: §4.2 · p. 9, Figure 3 · p. 10
  • DPO averages rise from 53.9 for the base policy to 57.9 with ReVeR (SFT); under GRPO, ReVeR (G-RL) leads at 57.4.ReVeR-trained policies have the best value in every instruction-following column; under DPO, SFT reaches 77.1 strict IFEval accuracy versus 73.9 with Rubric-ARM-8B.Source: §4.2 · p. 9, Table 3 · p. 9

4.3. Semantic Matching against Expert Rubrics

  • ReVeR (SFT) has the highest expert-rubric agreement on all six measures, with F1 of 32.1 on chat and 46.3 on instruction following.Rubric-ARM-8B, the closer baseline, reaches F1 of 23.6 and 34.9 on those subsets.Source: §4.3 · p. 10, Table 4 · p. 11
  • Generator RL lowers expert agreement as hard-rule shares fall from 48.2% to 37.8% on chat and from 66.1% to 52.5% on instruction following.On instruction following, some lost expert criteria are explicit binary requirements, while added principles name generic qualities such as accuracy.Source: §4.3 · p. 11, Table 4 · p. 11
  • On RubricBench, lower expert agreement leaves preference accuracy almost unchanged, while generator RL reduces ties and shifts rubrics toward principles.Pairs decided by principles are judged less accurately than pairs decided by hard rules, offsetting the gain from fewer ties.Source: §4.3 · p. 11

4.4. Case Study

  • Given the same hard rule, ReVeR’s program-based judge selects the compliant response in all six runs, while Rubric-ARM prefers the other response in all three repeats.The example uses a requirement to include at least 15 lowercase p’s; ReVeR counts them with a program.Source: §4.4 · p. 11

5. Conclusion

The conclusion presents ReVeR as a pointwise rubric reward model whose criteria come from the task and whose judgments use executed evidence. It reports improvements over pairwise rubric rewards while emphasizing that rubric fidelity, preference accuracy, and downstream utility are distinct properties.

5. Conclusion

  • ReVeR derives criteria by reasoning about the task, and its judge reaches each verdict from the response and evidence it executes.The paper presents ReVeR as a pointwise rubric reward model and ReVeR-Traces as traces for rubric generation and tool-augmented judgment.Source: §5 · pp. 11–12
  • Label-blind distillation trains both components, with preference labels filtering teacher outputs rather than changing rubrics or judgments.The conclusion describes ReVeR-Traces as a collection of reasoning traces for rubric generation and tool-augmented judgment trajectories.Source: §5 · p. 12
  • With 4B models, ReVeR has the highest average preference accuracy among single-judgment reward models and improves on pairwise rubric rewards across several evaluations.The conclusion names expert-rubric agreement, best-of-N selection, and policy optimization, and says fidelity, accuracy, and downstream utility must be measured separately.Source: §5 · p. 12

A. Distribution of Principle Scores

The score distributions show that principle scores vary broadly and carry preference signal, while also helping distinguish responses tied on hard rules. Generator RL changes both the number and average score of principles.

A. Distribution of Principle Scores

  • Preferred responses have 47% (SFT) and 33% (G-RL) of principle scores at 80 or above; rejected responses have 19% and 21% below 10.The mean score gap between preferred and rejected responses is 18.7 points for SFT and 15.9 for G-RL.Source: A. Distribution of Principle Scores · p. 19, Figure 5 · p. 20
  • On 41–42% of pairs tied on hard rules, the principle average favors the preferred response in 58–59%.The distributions overlap: 28% and 19% of scores for rejected responses are also 80 or above.Source: A. Distribution of Principle Scores · p. 19
  • G-RL rubrics contain 2.7 principles per response versus 1.9 for SFT, and their principles receive a lower mean score, 48.5 versus 55.5.The score distributions use more than 90 distinct values; 1.5% of SFT scores and 0.5% of G-RL scores equal 100.Source: A. Distribution of Principle Scores · p. 19

B. Training and Evaluation Details

The evaluation combines rubric-conditioned judging and online policy training with multiple selection and preference-optimization procedures. Training data, validation rules, baselines, and settings define the evaluated systems.

Judging protocol

  • Judging uses one rubric per prompt and one episode with three program calls, each subject to a three-second timeout; invalid judgments receive no benchmark score or policy reward.When calls are exhausted, unsupported required properties count as unsatisfied, and invalid verdict trajectories are resampled.Source: B. Training and Evaluation Details · p. 20

Distillation data

  • ReVeR-Traces pairs instructions with rubric reasoning and typed criteria, while judging records add a response, program trajectory, outputs, explanations, and scores.The acceptance test retains triples only after applying its one-pass teacher rubric-and-judgment procedure; among rejected triples, 74.2% reverse the preference, 19.5% tie, and 6.3% are invalid.Source: B. Training and Evaluation Details · pp. 20–21

Evaluation and selection

  • The comparisons include reasoning RMs, rubric-based RMs, tool-augmented judging, an untrained model under the same protocol, and the teacher.Best-of-N uses one rubric and scores each candidate in its own episode, returning the highest valid score.Source: B. Training and Evaluation Details · p. 21

Policy training data

Preference optimization

  • DPO pairs come from eight candidates per instruction, scored twice; the highest and lowest complete candidates are kept when their normalized-score gap exceeds 0.05.Pairwise baselines instead select best and worst candidates through a bubble pass of comparisons.Source: B. Training and Evaluation Details · p. 22

Online reinforcement learning

  • GRPO generates one rubric per prompt and scores every rollout against that shared rubric, putting group-relative comparisons on a common scale.Pairwise baselines use the online reward from their original work.Source: B. Training and Evaluation Details · p. 22

C. Behaviour of the Trained Policies

The policy trained with Rubric-ARM under GRPO shows a distinctive pattern of self-certification and instruction-following failures. The reported surface-behavior measures cover IFEval and MulDimIF responses.

Policy behavior

  • The Rubric-ARM GRPO policy ends 133 IFEval responses with a summary or self-assessment, 56 claiming compliance, and fails 63% of capitalization constraints.It also states compliance in 228 MulDimIF responses and is the only policy showing the pattern at this scale.Source: C. Behaviour of the Trained Policies · p. 22
  • 12.0 points of prompt-level strict IFEval accuracy return under the loose criterion, which tests responses without their first line, last line, or markdown emphasis.The reported comparison locates the failures in what the policy adds around its answer.Source: C. Behaviour of the Trained Policies · p. 22

Rubric behavior

  • Table 7 tracks criteria per rubric, rubrics without a principle, and preceding reasoning length; Table 8 decomposes precision change into criterion-type shares and match rates.The supplied passages identify these tables’ scopes but do not give their cell values.Source: Table 7 · p. 23, Table 8 · p. 23

D. Further Analysis of Generator RL

Generator RL changes rubric composition and criterion coverage, while preference accuracy relates differently to coverage across chat and instruction-following prompts. The analysis details these shifts and their scope.

Rubric shape

  • Rubrics retain nearly the same number of roughly 21-word criteria after RL, but almost all contain a principle and preceding reasoning falls to less than half its former length.The reported shape statistics do not depend on the criterion matcher.Source: D. Further Analysis of Generator RL · p. 23

Precision decomposition

  • 8.0 points on chat and 8.9 on instruction following: pooled precision falls after RL, mainly through less-matching principles on chat and a shift toward principles on instruction following.Hard-rule match rates contribute less than one point to the drop on either subset.Source: D. Further Analysis of Generator RL · p. 23

Position within the rubric

  • On chat, match rates for the first two rubric criteria stay about the same after RL, while the gap opens from the third; hard-rule match at position four falls from 47.7% to 34.3%.The second criterion is a principle in 53% of post-RL rubrics, up from 36%.Source: D. Further Analysis of Generator RL · p. 23, Table 9 · p. 24

Coverage of expert criteria

Lost criteria

  • Lost instruction-following criteria include explicit language, capitalization, separator, and limerick-format requirements; lost chat criteria concern direct answers and answer structure.Many lost instruction-following criteria have true-or-false outcomes.Source: D. Further Analysis of Generator RL · p. 24

Generic criteria

  • RL-added criteria include requests for explanations and clarifying questions; criteria with examples become more common while instruction-word overlap and match rates for relevance, tone, style, and clarity decline.Example-bearing principles rise from 39% to 62%, and example-bearing hard rules from 18% to 34%; relevance match falls from 32.7% to 19.8%.Source: D. Further Analysis of Generator RL · pp. 24–25

Preference accuracy

  • Instruction-following accuracy rises from 37.0% with no covered expert criteria to 72.0% with three, while chat accuracy stays between 45% and 58% across coverage levels.On prompts where G-RL F1 trails SFT by at least 15 points, accuracy is 60.7 versus 57.7 on chat and 55.6 versus 59.8 on instruction following.Source: D. Further Analysis of Generator RL · p. 25, Table 12 · p. 25

Example

  • For the prompt “how to drink water,” G-RL criteria stay on topic but are more detailed and include parenthetical examples; the matcher treats them as less equivalent to expert criteria.The example compares the two generators’ rubrics against expert criteria.Source: D. Further Analysis of Generator RL · p. 25

E. Case Studies

The case studies compare ReVeR and Rubric-ARM decisions on two RubricBench prompts and show one generator-level rubric comparison for a chat prompt.

Case studies

  • The appendix presents two RubricBench prompts on which ReVeR and Rubric-ARM reach opposite decisions.Instructions, responses, rubrics, programs, and judge outputs are quoted for the cases.Source: E. Case Studies · p. 25

Rubric comparison

  • For “how to drink water,” Table 13 compares expert criteria with each generator’s closest criterion, marking matches with checkmarks.SFT matches three expert criteria, while G-RL matches one.Source: Table 13 · p. 26
  • The table’s key comparison is how many expert criteria each sampled rubric matches: three for SFT and one for G-RL.The comparison concerns one sampled rubric from each generator.Source: Table 13 · p. 26

E.1. A Case Decided by the Judge

In this constrained-college example, ReVeR judges Response A correctly in all six runs, while the pairwise rubric judge repeatedly favors Response B. The judges apply the letter-count rule differently despite both rubrics containing it.

E.1. A Case Decided by the Judge

  • The example’s two rubrics both include the letter-p requirement, but the judges apply that shared rule differently.ReVeR’s rubric also lists hard rules and graded principles, including relevance, organization, tone, and concision.Source: §E.1 · pp. 26–27
  • ReVeR’s judge scores each response in its own episode, running one program per response; it counted 16 lowercase “p”s in Response A and 13 in Response B, with aggregate scores 3.10 and 2.00, respectively.Source: §E.1 · pp. 27–28
  • ReVeR prefers Response A and is correct in all six runs, while the pairwise judge prefers Response B across all six presentation-order runs.Response A asks for more information; Response B asks follow-up questions, and both fail the primary recommendation requirement.Source: §E.1 · p. 28
  • The pairwise judge treats Response B’s letter count as decisive, although the swapped-order output still selects the original Response B.The expert label identifies Response A as better.Source: §E.1 · pp. 26, 28

E.2. A Case Decided by the Rubric

In the savings example, ReVeR’s judge recognizes the missing interest-rate information and catches an incorrect calculation, preferring Response A correctly in all six runs. The pairwise judge instead favors a specific numerical answer, even though it is wrong.

E.2. A Case Decided by the Rubric

  • The instruction omits an interest rate, but Rubric-ARM requires multiplying an annual amount by years, whereas ReVeR’s rubric allows requesting missing parameters or stating assumptions.ReVeR’s rubric also requires correct calculations when given and asks for assumptions and uncertainties to be explained.Source: §E.2 · pp. 28–30
  • Response A asks for the interest rate, compounding frequency, and additional contributions before calculating the savings.Response B instead supplies a calculation based on stated assumptions.Source: §E.2 · p. 29
  • ReVeR identifies that 120,000 × 1.0330 equals 291,271.50, not 336,960.71, and prefers Response A correctly in all six runs.It scores Response B 0 on the rubric’s calculation-correctness rule.Source: §E.2 · p. 30
  • The pairwise judge prefers Response B’s specific numerical answer in the original order and still selects that original response after swapping presentation order.The calculation is incorrect: the response assumes 3% annual interest compounded annually and gives $336,960.71.Source: §E.2 · pp. 29–30

F. Prompts

The prompts define how ReVeR derives operational rubrics from instructions and evaluates responses independently, separating non-negotiable requirements from graded qualities through pointwise scoring, bounded tool use, calibrated guidance, and auditable JSON outputs.

Prompt setup

  • The generator prompt is used in distillation, generator RL, and evaluation; the inference judge prompt permits three tool calls, while teacher trajectories use one.Both prompts use RubricBench in-context examples excluded from reported RubricBench results; the judge receives the instruction, one response, and numbered criteria as JSON, and its Python tool has a three-second timeout.Source: F. Prompts · p. 31

Rubric generator prompt

  • The generator analyzes the task without drafting its answer and creates a concise, operational rubric focused on functional requirements rather than a fixed answer plan.It considers explicit and reasonably implied requirements, task-specific risks, and observable success conditions.Source: F. Prompts · p. 31
  • Criteria must be grounded, generalizable to different strong responses, useful for unseen responses, discriminative, and minimal; unsupported preferences and answer checklists are excluded.The prompt also cautions against treating length, examples, or generic qualities as inherently valuable.Source: F. Prompts · pp. 31–32
  • Hard Rules cover explicit non-negotiable requirements, while Principles grade implicit qualities; core factual errors are Hard Rules only when they invalidate the response.The prompt preserves modality: mandatory language and strict limits can create Hard Rules, but optional wording must not become mandatory.Source: F. Prompts · p. 32

Judge prompt

  • The judge evaluates one response against every rubric independently, without comparing it to another candidate or relying on an imagined alternative.It must return one judgment per criterion without skipping, merging, rewriting, splitting, reordering, or adding criteria.Source: F. Prompts · pp. 34–35
  • Hard Rules receive binary 0-or-1 scores with no partial credit, whereas Principles receive integer scores from 0 to 100 using qualitative anchors.The judge is instructed to inspect concrete evidence for each criterion and consolidate necessary deterministic checks into a tool call.Source: F. Prompts · p. 35

F. Prompts

  • The prompt allows at most three tool calls, preferring none for semantic judgments and one consolidated call for deterministic checks.Python is reserved for mechanical checks that could materially affect scores; semantic judgments should be made directly.Source: F. Prompts · pp. 35, 36, 39
  • Judges must assess only supplied rubrics and cannot reward verbosity, presentation, or added detail unless it materially improves criterion fulfillment.Brevity is not itself a weakness, and repetitive, generic, irrelevant, or unsupported additions do not strengthen coverage.Source: F. Prompts · p. 36
  • Principle scoring should reflect graded coverage and affirmative evidence of fulfillment, not require perfection for moderate scores or award near-perfect scores by default.Coverage ranges from superficial mention to near-ideal treatment; a score around 60 can represent useful but limited fulfillment, while near 100 requires affirmative evidence.Source: F. Prompts · pp. 36–37
  • Judges should weigh reasoning validity, clarity, actionability, related qualities, and the net effect of strengths and weaknesses against each Principle.For factual claims, unsupported or uncertain material claims lower the relevant score; a proven-false core claim essential to the deliverable caps the score at 40.Source: F. Prompts · p. 37
  • Each explanation must identify the rubric requirement, concrete evidence, material weaknesses or uncertainty, and why the evidence supports the score.The output must be one valid JSON object with one ordered judgment per supplied rubric and exactly the specified judgment fields.Source: F. Prompts · pp. 39–40

Related papers

Loading 2610.18766v1…