PaperlayerView exact revision on arXiv

Source-linked AI summary

Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics

Maoqi Liu, Junwei He, Bowen Zhang, Feiran Li, Wentao Ma, Rongyi Lin, Shuhan Zhong, Quan Fang

arXiv:2609.38847v1cs.LGcs.AI

TL;DR

In Rubric-RL, weighted sums can let easy criteria compensate for a missed critical decision, raising rubric coverage while clinical appropriateness on held-out physician criteria falls below the untrained model. The paper introduces ProRubric, which groups unchanged criteria into jointly satisfied protocol-level dimensions. It reports higher appropriateness without losing coverage and the best seven-benchmark average at both scales.

  • Problem

    Weighted-sum rubric rewards allow criteria to compensate for one another, and clinical policies can gain coverage while appropriateness on held-out physician criteria falls below the untrained model.

    Source: §1 · p. 2

  • Method

    ProRubric groups rubric criteria offline into protocol-level dimensions that count only when all criteria hold and the failure clause does not fire.

    Source: §1 · p. 2

  • Results

    10.8 points: ProRubric raises appropriateness without losing coverage and has the best seven-benchmark average at both model scales.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Reward validity depends on aggregation as well as on what a rubric verifies; with criteria fixed, changing aggregation moves the collapse in medicine and dialogue, while science also depends on wording.

    Source: §7 · p. 10

    The work evaluates single-turn responses with an LLM judge calibrated against physicians, rather than clinical trials; interactive dialogues and human studies remain future work.

    Source: §7 · p. 10

Abstract

from arXiv · show

Rubric-based reinforcement learning (Rubric-RL) trains language models where no verifier exists. A judge checks each criterion of a rubric, and the verdicts are aggregated into a reward, most often by a weighted sum. We show that this additive aggregation is the weak point. Under a sum, criteria compensate for one another: a policy that misses the one decision that matters can buy the points back with advice nobody asked for. On clinical consultation, such a policy scores higher and answers worse. Rubric coverage rises while appropriateness on held-out physician criteria falls below the untrained model. The medical criteria are not to blame. Grouped so that they must hold together, the same criteria, unchanged to the word, recover a third of the loss; shorter answers recover almost none. We therefore propose Protocol-level Rubrics (ProRubric), which keeps what the criteria ask for and changes how they are aggregated. It groups a checklist into a few protocol-level dimensions. A dimension counts only when all of its criteria hold and its failure clause does not fire. The grouping is done once, offline, and leaves the optimizer unchanged. ProRubric raises appropriateness by 10.8 points without losing coverage and has the best seven-benchmark average at both scales. Reward validity is set not only by what a rubric verifies, but by how it aggregates. Code is available at https://github.com/Estrellajer/ProRubric

1 Introduction

Rubric-RL’s weighted-sum aggregation lets easy criteria compensate for missed critical decisions, so coverage can rise while clinical appropriateness falls. ProRubric changes aggregation while keeping criteria fixed, improving appropriateness without sacrificing coverage.

1 Introduction

  • Additive aggregation lets a policy that misses a critical decision recover reward through easier criteria and unasked-for material.In the fever example, the trained policy produces 10.9k characters and fails both physician-written criteria.Source: §1 · p. 2
  • Coverage rises from 26.6 to 32.1 while appropriateness on held-out physician criteria falls from 54.1 to 26.0, below the untrained model.A judge given no rubric also prefers the untrained model.Source: §1 · p. 2
  • Grouping verbatim criteria conjunctively recovers a third of the appropriateness loss, while holistic scoring of the list recovers nearly 40%.Shortening answers by 42% leaves appropriateness unchanged; in medicine and dialogue, aggregation changes the collapse with criteria fixed, while in science both grouping and synthesis matter.Source: §1 · p. 2
  • 10.8 points: ProRubric raises appropriateness without losing coverage and achieves the best seven-benchmark average at both model scales.It groups criteria offline into protocol-level dimensions that count only when all criteria hold and the failure clause does not fire.Source: §1 · pp. 2–3
  • Across seeds, scales, and judges, additive Rubric-RL raises coverage while held-out physician-criteria appropriateness falls below the untrained model.The standard score records coverage but not this reversal.Source: §1 · p. 2
  • With criterion text fixed, changing aggregation moves the collapse in medicine and dialogue, and partly in science; the incentive is readable from the reward before training.The analysis finds that length, KL, weights, judge, and data do not account for the result.Source: §1 · p. 3

2 Preliminaries

Rubric rewards depend both on which criteria are checked and how their verdicts become a scalar. The preliminaries distinguish additive scoring and compensation from the two medical evaluation axes used to assess outcomes.

2.1 Rubric Rewards: Criteria and Aggregation

  • A rubric pairs natural-language requirements with weights; standard explicit aggregation scores criteria separately and combines verdicts as a normalized weighted sum.Alternative aggregations can score conjunctive groups or the whole checklist, while holding the criteria and optimizer fixed.Source: §2.1 · p. 3
  • Under additive scoring, satisfying easier criteria can recover reward lost by missing a critical criterion, creating an incentive for reward hacking.The paper optimizes with GRPO and frames this as an objective-level argument, not a convergence proof.Source: §2.1 · p. 3

2.2 Two Axes: Rubric Coverage and Appropriateness

  • Medical policies are assessed on rubric coverage under example-specific criteria and appropriateness under physician-consensus criteria held out from training.Both axes are scored by DeepSeek-V4-Pro, so the paper calls the latter appropriateness rather than clinical utility.Source: §2.2 · p. 3
  • The study asks whether changing aggregation alone, with criteria and optimizer fixed, can make appropriateness rise alongside coverage.The preliminaries preview that explicit aggregation raises coverage while appropriateness falls below the untrained model.Source: §2.2 · p. 3

3 The Reward-Hacking Channel in Additive Aggregation

With additive aggregation, clinical Rubric-RL raises coverage while reducing appropriateness on held-out physician criteria. Fixed rewards for individual criteria let policies earn points for easy additions instead of the decision that matters.

3.1 Scoring Higher, Answering Worse

  • Coverage rises while appropriateness falls below the untrained model across scales and seeds; rubric-free judges also prefer the untrained answers.The comparison is on clinical consultation; the exploration baseline using the same reward falls just as far.Source: §3.1 · pp. 3–4
  • The same responses score higher on coverage criteria and lower on physician criteria, indicating that the checked criteria—not the judge—set the sign of the gap.A judge closer to the training reward understates the damage rather than creating it.Source: §3.1 · p. 4

3.2 Why the Cheap Behavior Is the Rational One

  • Under additive aggregation, every criterion pays a fixed rate, making cheap criteria rational targets even when a correct triage decision requires harder reasoning.The passage frames this as an argument about the objective, not a convergence proof about the optimizer.Source: §3.2 · p. 4

3.3 Reading the Incentive off the Reward

  • Reward-change tests show that Rubric-RL pays for naming concrete items, leaves unwarranted high-risk tests cost-free, and barely penalizes deleting a critical recommendation.Grouping the same criteria verbatim stops paying for a name, while ProRubric failure clauses make the needless test costly.Source: §3.3 · pp. 4–5

4 Method: Protocol-Level Rubrics (ProRubric)

ProRubric replaces fixed per-criterion aggregation with offline protocol-level dimensions that must hold together. Failure clauses prevent a dimension from counting when a phase is undermined, while rewards remain additive across dimensions.

4 Method: Protocol-Level Rubrics (ProRubric)

  • ProRubric changes how criteria are aggregated, not what they ask for, and writes protocol phases into rubrics offline without changing the optimizer or served policy.It treats protocols as gated rather than tallied.Source: §4 · p. 5

4.1 Conjunctive Dimensions

  • Unlike fixed per-criterion rewards, a protocol dimension counts only when all its criteria hold jointly; penalty criteria act as hard non-violation constraints.The grouped criteria carry their aggregated weight when the dimension is satisfied.Source: §4.1 · p. 5
  • The reward is a normalized weighted sum over dimension verdicts, which the judge verifies directly using synthesized protocol prompts.Source: §4.1 · p. 5
  • Under an idealized independence assumption, a criterion pays in proportion to the likelihood that the rest of its dimension holds.A fragment in a failing dimension earns almost nothing, while the last missing piece of a nearly complete dimension earns the full dimension weight.Source: §4.1 · p. 5

4.2 Failure Clauses

  • Each dimension ends with a failure clause that voids it when a named condition, such as delaying urgent care, occurs despite other criteria being met.The judge reads both the conjunction and failure clause and returns one verdict per dimension.Source: §4.2 · p. 5
  • A fired failure clause cannot be offset within its dimension, but reward remains additive across dimensions, reducing compensating units from m criteria to K dimensions.The approach reduces rather than eliminates compensation.Source: §4.2 · p. 6

4.3 Generating the Dimensions

  • A generator groups criteria, writes each dimension description with a failure clause, assigns summed weights, and validates that every criterion belongs to exactly one dimension.The studied protocols use 2 ≤ K ≤ 5 dimensions; generation runs once per prompt before training and adds no serving cost.Source: §4.3 · p. 6

5 Experiments

The experiments find that ProRubric improves the target-domain trade-off and leads on average across seven benchmarks, while changing only aggregation moves appropriateness collapse in several domains. The results also identify residual failures and show that the reward effects depend on what criteria pay for.

5.1 Experimental Setup

  • Experiments train Qwen3-4B and Qwen3-8B policies across four domains and evaluate seven benchmarks, with medical appropriateness measured on held-out physician-consensus criteria.The setup uses three seeds for main-table trained methods; responses are evaluated without a rubric at inference.Source: §5.1 · p. 6

5.2 Main Results

  • At 4B, ProRubric reaches 36.8 appropriateness against Rubric-RL’s 26.0 and expands coverage to 35.0; it also leads the seven-benchmark average at both scales.Its 4B average margin over Rubric-RL is +1.3 points, with a 95% interval of [+0.2, +2.4]; it is ahead of both RL baselines on every paired seed at both scales.Source: §5.2 · pp. 6–7, Table 2 · p. 7

5.3 Changing Only the Aggregation Moves the Collapse

  • With criterion text fixed, changing aggregation moves appropriateness upward; raw-AND recovers most of ProRubric’s medical gain, while verbatim implicit aggregation matches within confidence intervals.Raw-AND has higher variance (±3.6 vs ±2.7) and lower coverage (33.8 vs 35.0); both controls reproduce at 8B over three seeds.Source: §5.3 · p. 7, Table 3 · p. 7
  • Length matching, reweighting, and a stronger KL penalty do not explain the collapse; the KL penalty restores appropriateness only by stopping learning.The 42%-shorter variant leaves appropriateness unchanged, while at β=0.04 both rewards approach the untrained model on both axes and in length.Source: §5.3 · p. 8

5.4 When the Collapse Comes, and Where ProRubric Stops

  • In retrained medical models, Rubric-RL appropriateness falls from 55.8 to 23.8, while ProRubric ends at 35.3; without per-criterion credit, collapse is slower but not stopped.At step 50, Rubric-RL is at 35.9 and ProRubric at 50.3; at every checkpoint every Rubric-RL seed is below every ProRubric seed.Source: §5.4 · p. 8
  • ProRubric remains below the untrained model on appropriateness because added material still pays across dimensions, though its additions are more substantive than Rubric-RL’s.Responses average 9.8k characters versus 3.0k for the untrained model, and added content costs 17–38 redundancy-subscore points.Source: §5.4 · p. 8

5.5 What a Rubric Pays For Decides Whether a Domain Collapses

  • Across medicine, science, and dialogue, explicit aggregation lowers appropriateness, while writing improves; the reported changes are −28.9, −11.2, −7.1, and +16.0, respectively.The analysis uses three-seed means on 200 prompts per domain.Source: §5.5 · p. 9
  • Mention-style criteria permit hollow naming, while substantive science criteria require elaboration; why writing improves remains open.Naming edits leave the science reward unchanged in 195 of 300 cases; adding relevant unrequested substance earns reward in every domain and costs more redundancy points than naming.Source: §5.5 · p. 9

6 Related Work

Prior work uses rubrics for evaluation, reward, exploration, and training, while research on aggregation has proposed alternatives to weighted sums. This paper focuses on how aggregation affects reward validity when criteria are held fixed.

Rubrics as rewards

  • Rubrics became a standard LLM-evaluation interface and now support checklist rewards, rubric datasets, rollout scaffolding, trajectory selection, and rubric-guided self-distillation.These approaches change what criteria ask or where they enter training; their rubric rewards still use weighted sums.Source: §6 · p. 9

Aggregating the criteria

  • Aggregation alternatives include implicit whole-checklist scoring, prerequisite graphs, criterion dropout, and attributing rubric items to reasoning steps.ProRubric instead groups a plain checklist into jointly satisfied dimensions, without annotated relations, and tests performance on criteria withheld from training.Source: §6 · p. 9

Reward hacking

  • Reward hacking is documented in RLHF and DPO, including length bias; prior rubric-RL work reports rubric-based versus rubric-free judge reversals.This paper traces the reversal to aggregation changes in medicine and dialogue while holding the judge and criterion text fixed.Source: §6 · p. 10

7 Conclusion

The conclusion argues that rubric reward quality depends on aggregation as well as verification. ProRubric groups criteria offline into jointly satisfied dimensions and improves appropriateness without losing coverage or changing the optimizer.

7 Conclusion

  • A weighted sum lets cheap criteria compensate for a missed critical one, raising coverage while held-out physician-criteria appropriateness falls below the untrained model.Changing only aggregation moves the collapse in medicine and dialogue; in science, wording matters as well.Source: §7 · p. 10
  • Clinical evaluation uses single-turn responses scored by an LLM judge calibrated against physicians, not clinical trials; interactive dialogue and human studies remain future work.The medical benchmarks are not evidence of clinical validation.Source: §7 · p. 10

Ethics Statement

The medical experiments use public benchmarks and no private data or human subjects; the trained models are research artifacts, not validated for clinical use.

Ethics Statement

  • The models are not validated for clinical use and should not be used for medical advice or patient triage.The experiments use public benchmarks and involve no private data or human subjects.Source: Ethics Statement · p. 10

AI Use Statement

The statement describes language-model assistance and author verification, while the appendix roadmap and methods document data, rubric generation, and structural validation.

AI Use Statement

  • Language models generated ProRubric dimensions and served as judges; they also helped polish prose and format LaTeX, with authors verifying every claim.Source: AI Use Statement · p. 10

Appendix Roadmap

  • The appendix covers reproducibility details, complete empirical evaluations, sensitivity analyses, and qualitative examples.Source: Appendix Roadmap · p. 14

A.1 Data Sources and Training Datasets

  • Training uses domain-specific RubricHub subsets and RaR-Science, with prompts deduplicated or filtered and evaluations conducted out of domain.Rubric-RL and ProRubric use identical prompts, so their observed differences come from reward aggregation rather than data curation.Source: §A.1 · p. 15

A.2 Rubric Generation Prompts and Protocol Templates

  • Offline generation groups atomic rubric items into two to five protocol dimensions, rewrites each group, and assigns summed item weights.The generator sees the atomic rubric but no evaluation criterion; generation runs before training.Source: §A.2 · p. 15

A.3 Validation, Repair, and Filtering

  • Validation checks that each criterion is assigned exactly once and filters rubrics that fail structural requirements, but does not verify description accuracy or semantic coverage.Source: §A.3 · p. 16

A.4 Training Configurations and Baseline Differences

The training configurations align key settings across methods, while the ablations vary aggregation, rubric wording, and added criteria.

A.4 Training Configurations and Baseline Differences

  • Rubric-RL and ProRubric share optimizer and rollout settings; the main training difference is the rubric, and overlong-response shaping applies across rubric-reward runs.The shaping penalty begins beyond 4,096 tokens and is capped at 0.5; validation scores are unshaped.Source: §A.4 · p. 16, Table 4 · p. 17
  • In writing and dialogue, RuscaRL uses its last saved checkpoint at step 250 after exhausting its training set, while OPSD is evaluated at different later steps.SFT uses a mixed-domain corpus and one model per seed, whereas RL policies are trained per domain.Source: §A.4 · pp. 16–17

A.5 Ablation Construction

  • Ablations vary joint satisfaction, graded scoring, dimension count, failure clauses, an added appropriateness criterion, implicit scoring, response limits, KL loss, criterion weights, and generator.The added appropriateness criterion is independently scored, so failure on it does not zero other criteria.Source: §A.5 · p. 17

B.1 Benchmark Scoring and Judge Configurations

The section specifies benchmark scoring and judge comparisons, then describes paired evaluation sets and rubric-free response comparisons under length controls.

B.1 Benchmark Scoring and Judge Configurations

  • Benchmarks use distinct scoring procedures, including signed-weight criterion coverage for HealthBench and five-level rubric coverage for ResearchQA.HealthBench divides satisfied signed weights by total positive weights before averaging over questions.Source: §B.1 · p. 18
  • DeepSeek-V4-Pro scores 0.62 balanced F1 against physician labels, compared with 0.65 for the average physician and 0.71 for GPT-4.1.The shared judge bounds how well appropriateness tracks physicians, not the comparison between methods.Source: §B.1 · p. 18

B.2 Paired Evaluation Sets and Statistical Estimation

  • Model comparisons use paired intersections of questions scored without evaluator failure, with sample sizes varying by comparison.Rubric-free pairs are judged in both orders; reported conditional win rates exclude ties and inconsistent decisions, and equal-budget responses share a 2,048-token ceiling without equalizing realized lengths.Source: §B.2 · p. 18, §B.3 · p. 19

C.1 Per-Domain and Per-Seed Results

The reported comparisons span per-seed results, reward-structure sensitivity, judge checks, training dynamics, and a fixed appropriateness criterion. Mean gains coexist with substantial variation across individual benchmarks.

C.1 Per-Domain and Per-Seed Results

  • The mean improvement is 10.8 points under DeepSeek-V4-Pro and 6.1 under Doubao-lite across three seeds; single-benchmark differences vary substantially across seeds.On GPQA-Diamond the paired differences range from −5.6 to +6.6, while the HealthBench difference keeps its sign across seeds.Source: §C.1 · pp. 19–20

D.1 Sensitivity Analysis: All Reward Structures

  • In the writing rubric, naming an item earns +1.3 under explicit aggregation, an interval containing zero.That rubric was tested with two naming perturbations because the other three perturbations are clinical.Source: §D.1 · p. 21

D.2 Notes on Section 5

  • Judge choice changes some ablation comparisons: Doubao-lite prefers longer variants, while the differences shrink roughly three- to fourfold under DeepSeek-V4-Pro.The two judges also differ in their appropriateness and coverage verdicts on the same responses.Source: §D.2 · p. 21
  • At 8B, implicit aggregation reaches 47.6 ± 1.0 appropriateness against ProRubric’s 45.8 ± 1.8 and Rubric-RL’s 29.5 ± 1.1; raw-AND recovers 11.4 of ProRubric’s 16.3 points.At 4B, the K=1 result varies by seed and has lower average coverage; changing the dimension generator shifts appropriateness by less than two points.Source: §D.2 · pp. 21–22
  • At KL β=0.04, ProRubric’s appropriateness advantage over Rubric-RL is +4.3 points, below the preregistered +5 threshold; its score remains below the untrained model.In retrained runs, step-300 appropriateness is 35.3 for ProRubric and 23.8 for Rubric-RL; nearly complete dimensions are completed more often than partly or wholly unmet ones.Source: §D.2 · pp. 22–23
  • SFT exceeds the untrained model on consensus criteria but incurs out-of-domain losses; with four domains, task and rubric-shape effects cannot be separated.On HealthBench, SFT is level with ProRubric at both scales.Source: §D.2 · p. 23

D.3 A Fixed Appropriateness Criterion

  • Appending the fixed appropriateness criterion raises appropriateness under each aggregation without reducing coverage, but the criterion can flag content-preserving paraphrases.The judge marks it unmet on 56.3% of content-preserving paraphrases, and adjudication finds many unmet verdicts unwarranted.Source: §D.3 · pp. 23–24

D.4 Cross-Domain Appropriateness Evaluation

Cross-domain evaluation finds that Rubric-RL can win whole-answer preferences while losing on appropriateness, and that added material—not its placement—accounts for most of the charge at matched budgets. ProRubric preserves the same requirements through conjunctive protocol dimensions, with a coverage trade-off on exhaustive information-retrieval tasks.

D.4 Cross-Domain Appropriateness Evaluation

  • In science, DeepSeek-V4-Pro preferred Rubric-RL in 262/300 pairs, yet appropriateness fell 12.0–13.0 points and redundancy passed only 0–1% versus 35–42% for the untrained model.Dialogue showed the same preference/appropriateness pattern: both judges preferred the trained policy, whose appropriateness fell 7.1 points, mostly on redundancy.Source: §D.4 · p. 25

D.5 Where Added Material Sits: Appending versus Integrating

  • At common budgets, the 31-point cross-domain spread disappeared and no domain charged nothing; placement was at most second-order, but only 12–36% of prompts admitted both rewrites.The matched subsets contained 35–108 items per domain, with wide intervals for per-domain gaps.Source: §D.5 · pp. 25–26

E.1 ProRubric Rubrics and Representative Responses

  • For the car-accident example, 32 atomic criteria become four weighted protocol dimensions; each counts only if all its requirements hold and no failure clause fires.The four dimension weights sum to 233, exactly the total weight of the original atomic criteria.Source: §E.1 · p. 26

E.2 Failure Cases and Coverage Trade-offs

  • Under exhaustive-coverage scoring, ProRubric scored 61.1 vs. 62.9 at 4B and 64.9 vs. 65.8 at 8B, trading some coverage breadth for denser communication.The trade-off arose on literature-synthesis queries where the rubric tested exhaustive topical coverage and ProRubric sometimes omitted niche cases.Source: §E.2 · p. 26

Related papers

Loading 2609.38847v1…