MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning

1 INTRODUCTION

• Rubric-based RL assigns partial credit to instance-specific response criteria, but judges can credit information or actions absent from the response.This can favor responses that fail important requirements.Source: §1 · p. 1
• Vacuous Credit occurs when a judge awards a high criterion score despite the response omitting required information or an action.A single erroneous award can flip a response’s GRPO advantage and reverse its contribution to the policy update.Source: §1 · pp. 1–2
• Figure 1 contrasts credit for an unanswered query with MetaRubric’s joint improvement of the policy and rubric.Its inner loop optimizes evidence-aware responses, while its outer loop reweights criterion groups and revises criterion descriptions.Source: Figure 1 · p. 2
• MetaRubric alternates evidence-aware policy optimization with rubric adaptation, pairing each prompt with a counterfactual that changes one task-relevant fact.Credit is bounded by satisfaction, requirement coverage, and evidential support; stage-boundary rubric updates adapt criterion-group weights and revise descriptions while preserving the initial rubric’s requirements.Source: §1 · p. 2
• MetaRubric improves PubMedQA accuracy by 6.00–20.40 points across baselines, with further gains on HealthBench-Hard, MMOral-X, and MMOral-OPG.The reported ranges are 2.46–3.19 points on HealthBench-Hard, 1.29–2.13 on MMOral-X, and 3.09–3.82 on MMOral-OPG.Source: §1 · p. 2

2 RELATED WORK

• Rubrics as Rewards extends on-policy RL to open-ended tasks with instance-specific criteria; RubricHub and OpenRubrics scale criterion generation, while RRD refines granularity and weights.Other approaches use pairwise adaptive criteria, policy self-grading, or joint optimization of rubric generators with reasoners or judges.Source: §2 · p. 2
• Unlike approaches that focus on rubric generation or reward-hacking mitigations, MetaRubric couples evidence-aware policy optimization with response-guided rubric adaptation.Policy errors drive criterion revision on original and counterfactual cases and criterion-group reweighting, while the initial rubric’s intended requirements are preserved.Source: §2 · pp. 2–3
• Prior reward-hacking work studies rubric discrimination, verifier failures, deterministic verification, and random criterion dropout; MetaRubric identifies Vacuous Credit as a distinct judgment error.Its content-deletion tests and recorded-group replay connect awards surviving evidence removal with reversals in GRPO advantage sign.Source: §2 · p. 3

3 VACUOUS CREDIT IN RUBRIC-BASED REINFORCEMENT LEARNING

• Training maximizes expected judge reward over the prompt distribution, under the assumption that higher judge rewards reflect better rubric fulfillment.The section tests that assumption empirically and examines how erroneous credit affects the GRPO learning signal.Source: §3 · p. 3

3.1 REWARD GROWTH REVEALS VACUOUS CREDIT

• Training reward rose while independently audited rubric satisfaction declined overall, and auditor-flagged Vacuous Credit persisted throughout training.The evaluation used held-out responses assessed by three stronger judges, with criterion satisfaction counted only under unanimous agreement.Source: §3.1 · pp. 3–4
• Vacuous Credit is a high criterion score despite the response omitting required information or an action.Subtracting its reward component removes credit for fully omitted requirements while leaving other criterion contributions unchanged.Source: §3.1 · p. 4

3.2 VACUOUS CREDIT PERSISTS AFTER REQUIRED EVIDENCE IS REMOVED

• The deletion test compares paired edits that remove required content with control edits that remove generic statements while preserving the requirement.It tests whether criterion credit depends on the evidence needed to satisfy the target criterion.Source: §3.2 · p. 4
• 82.0% and 69.8% of target-criterion awards remained after required-content deletion for GPT-4o-mini and GPT-5.4-mini, respectively.The table reports retention rates in the deletion test.Source: Table 1 · p. 4, §3.2 · p. 4

3.3 IMPLICATIONS FOR POLICY OPTIMIZATION

• Vacuous Credit can make an omission earn the same or greater reward than a response that performs the required action, changing which responses GRPO reinforces.Repeated awards can lead policy updates to reinforce generic statements instead of rubric-required information or actions.Source: §3.3 · p. 4
• MetaRubric alternates N evidence-aware GRPO updates with rubric adaptation, carrying accepted rubric updates into the next stage.The figure presents this as an inner policy-optimization loop and an outer rubric-adaptation loop.Source: Figure 3 · p. 5

4 METARUBRIC: LEARNING TO REWARD FOR RUBRIC-BASED RL

• MetaRubric alternates GRPO policy updates under a fixed rubric configuration with stage-boundary adaptation using collected responses and held-out validation cases.The adaptation step updates weights from policy errors and accepts criterion revisions only when they pass validation.Source: §4 · p. 5

4.1 EVIDENCE-AWARE POLICY OPTIMIZATION

• Each prompt is paired with a counterfactual changing one task-relevant fact, and each response is scored against the rubric for its own prompt facts.Correspondence labels mark whether requirements are invariant or change in target, weight, or applicability across the pair.Source: §4.1 · pp. 5–6
• Criterion credit is the minimum of satisfaction, requirement coverage, and evidential support, so missing or unsupported required content limits the score.A response giving an incorrect requested value may cover the information slot without satisfying the criterion.Source: §4.1 · p. 6
• The rubric reward includes response-level support and quality checks, while an auxiliary QA reward independently tests whether prompt-specific facts are recoverable from responses.GRPO optimizes the combined reward over original and counterfactual prompts.Source: §4.1 · p. 6

4.2 ERROR-GUIDED WEIGHT ADAPTATION

• Criteria sharing severity class and correspondence label form groups whose weights shift according to their error rates relative to the mean.A separate grading pass measures errors, with group error aggregated across criteria, responses, and prompt-pair cases.Source: §4.2 · pp. 6–7
• Weight constraints preserve severity ordering, bound each group adjustment, and center adjustments at zero so adaptation changes relative emphasis rather than overall reward scale.This keeps a frequently missed lower-severity criterion from outweighing a must-have.Source: §4.2 · p. 7

4.3 SEMANTICALLY ANCHORED RUBRIC REVISION

• Rubric revisions clarify ambiguous or mismatched criterion descriptions against the original rubric’s anchored requirements and prompt facts, and require held-out agreement gains plus semantic review.If validation or semantic review fails, the current description is retained while updated weights carry over.Source: §4.3 · p. 7

5.1 SETUP

• Experiments cover PubMedQA and HealthBench-Hard for text-only medical QA, plus MMOral-X and MMOral-OPG for multimodal diagnosis and visual QA.Reported metrics are PubMedQA accuracy, HealthBench-Hard accuracy-axis score, MMOral-X mean score, and MMOral-OPG overall score; GPT-5.5 judges model-based test-set evaluations.Source: §5.1 · p. 7

5.2 MAIN RESULTS

• MetaRubric improves over static-judge GRPO in all 21 model-family–metric comparisons and achieves the highest score among training methods in 19.PubMedQA accuracy rises by 6.00, 6.80, and 20.40 percentage points for Qwen3-4B, Qwen3-8B, and Gemma-e2b, respectively; gains on the three open-ended benchmarks range from 1.29 to 3.82 points.Source: §5.2 · pp. 7–8, Table 2 · p. 8
• Auxiliary QA accuracy improves by 3.23–5.68 percentage points over GRPO across all three model families and open-ended benchmarks.The reader recovers more task-relevant information from responses, including in two settings where Dr. GRPO scores slightly higher on the benchmark.Source: §5.2 · p. 8

5.3 ABLATION STUDIES

• Freezing both criterion descriptions and weights causes the largest ablation drops: 3.17 points on HealthBench-Hard and 4.40 on MMOral-OPG.Revision has the larger individual effect on HealthBench-Hard, while weight adaptation matters more on MMOral-OPG; removing evidence checks reduces scores by 1.22 and 2.59 points.Source: Figure 4 · p. 8, §5.3 · pp. 8–9

5.4 DEEP ANALYSIS

• Rubric revision affects 45.2% of HealthBench criteria, 29.8% of PubMedQA criteria, and 60.6% of MMOral-RL criteria.Average descriptions grow by 5.5, 3.9, and 5.6 words, respectively; weight-only updates are excluded from revision counts.Source: Table 3 · p. 9, §5.4 · p. 9
• Retraining with the final rubric recovers only 5.0% of MetaRubric’s HealthBench-Hard gain and 20.7% of its MMOral-OPG gain.Final-rubric retraining scores 9.04 versus MetaRubric’s 13.02 on HealthBench-Hard, and 21.78 versus 26.35 on MMOral-OPG.Source: Table 4 · p. 9, §5.4 · p. 10
• The case study illustrates distinct controls: criterion wording specifies supported claims, while group weights change the relative influence of errors.In the example, the evidence criterion shifts from +6 to +5 and the overstatement penalty from −6 to −8.Source: Figure 5 · p. 9, §5.4 · p. 10

• MetaRubric combines evidence-aware scoring with response-guided criterion revision and criterion-group weights adapted to observed policy errors.The conclusion characterizes Vacuous Credit as high criterion scores despite absent required information or actions, with erroneous awards capable of reversing a response’s GRPO advantage.Source: §6 · p. 10

• The appendix includes sections on limitations, implementation and experimental details, prompt templates, and auxiliary question examples.Source: A APPENDIX · p. 13

• Each training configuration uses one seed, so the reported results do not quantify variation across training runs.Source: §A.1 · p. 13
• Many related works have not released training code, preventing direct comparisons under matched experimental settings.Source: §A.1 · p. 13

A.2.1 DEPLOYMENT DETAILS

• PubMedQA training uses the non-test PQA-L subset, with label-stratified validation and the official test set reserved for evaluation.Source: §A.2.1 · p. 13
• PubMedQA rubrics use six criteria: three positive criteria reward correct decisions, abstract evidence, and relevance, while three negative criteria penalize unsupported claims, contradictions, and generic rationales.Source: §A.2.1 · p. 13
• HealthBench training and validation use conversations outside HealthBench-Hard, while initial rubrics use official physician-written criteria and signed point values.Source: §A.2.1 · p. 13
• MMOral-X and MMOral-OPG share MMOral-RL, which contains 980 original examples and 8,672 criteria, averaging 8.85 criteria per example.Source: §A.2.1 · p. 13

A.2.2 AUXILIARY QUESTION CONSTRUCTION

• Auxiliary questions target distinct source facts or coherent fact groups; generation creates four-option items, and content review checks support, answer uniqueness, plausible distractors, and complementarity.Source: §A.2.2 · pp. 13–14
• Questions are screened without the response using Qwen3-1.7B, and failing items may be revised while preserving their target, correct answer content, and option position.Source: §A.2.2 · p. 14
• Auxiliary QA accuracy uses exact option matching on accepted tasks, counts invalid outputs as incorrect, and is distinct from point-weighted training score RQA.Source: §A.2.2 · p. 14
• Evaluation banks contain 4,117 HealthBench-Hard questions, 828 MMOral-X questions, and 981 MMOral-OPG questions; MMOral banks cover all source entries.Source: §A.2.2 · p. 14

A.2.3 REWARD COMPUTATION AND JUDGE PROMPTS

• Fulfillment scores align judge assessments in the same direction, and the minimum assessment caps criterion credit before weighted criterion contributions are summed.Source: §A.2.3 · p. 15

A.2.4 BASELINES AND TRAINING CONFIGURATIONS

• GRPO groups each prompt with its counterfactual, scores responses under their own prompt’s rubric, and standardizes rewards across the combined group.Source: §A.2.4 · p. 15
• At stage boundaries, criterion-group weights are updated from sampled response errors, while rubric revisions require validation agreement gains and semantic approval.Source: §A.2.4 · p. 15

A.2.5 EVALUATION PROTOCOLS AND METRICS

• Evaluation reports PubMedQA accuracy, HealthBench-Hard’s accuracy-axis score, MMOral-X’s three-subset mean, and MMOral-OPG’s overall score.Source: §A.2.5 · pp. 15–16