PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, Sean Hendryx

arXiv:2507.17746v2cs.LGcs.AIcs.CL

TL;DR

Real-world reasoning lacks the binary correctness signals that make RLVR effective. RaR uses instance-specific rubrics as on-policy rewards and outperforms Likert-based baselines across medical and science evaluations, with gains up to 31% on HealthBench and 7% on GPQA-Diamond.

  • Problem

    Extending RLVR to real-world reasoning remains underexplored because nuanced, multi-criteria judgments lack easily verifiable answers and preference models can capture superficial artifacts.

  • Method

    RaR uses instance-specific, checklist-style rubrics as structured, interpretable reward signals for on-policy reinforcement learning.

  • Results

    RaR-trained policies outperform Likert-based baselines across rubric-based and multiple-choice evaluations, with relative gains up to 31% on HealthBench and 7% on GPQA-Diamond.

  • Takeaways & Limitations

    Rubric-based rewards support strong cross-domain performance, improve alignment for smaller judges, and reduce dependence on large judge models.

  • Takeaways & Limitations

    Validation is limited to medicine and science, leaving broader dialogue, tool-use, and agentic-task evaluation for future work.

Abstract

from arXiv · show

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for complex reasoning tasks with clear correctness signals such as math and coding. However, extending it to real-world reasoning tasks is challenging, as evaluation depends on nuanced, multi-criteria judgments rather than binary correctness. Instance-specific rubrics have recently been used in evaluation benchmarks to capture such judgments, but their potential as reward signals for on-policy post-training remains underexplored. We introduce $\textbf{Rubrics as Rewards}$ (RaR), an on-policy reinforcement learning method that extends RLVR beyond verifiable domains by using rubric-based feedback. Across both medical and science domains, we evaluate multiple strategies for aggregating rubric feedback into rewards. The best RaR variant achieves relative improvements of up to $31\%$ on HealthBench and $7\%$ on GPQA-Diamond over popular LLM-as-judge baselines that rely on direct Likert-based rewards. These results demonstrate that RaR-trained policies adapt well to diverse evaluation formats, performing strongly on both rubric-based and multiple-choice tasks. Moreover, we find that using rubrics as structured reward signals yields better alignment for smaller judges and reduces performance variance across judge scales.

1. Introduction

Rubrics as Rewards (RaR) extends on-policy RL beyond domains with automatically verifiable answers by using instance-specific, checklist-style rubrics as structured reward signals. The framework targets interpretable, scalable supervision for expert reasoning and real-world tasks while improving performance across rubric-guided and verifiable evaluations.

  • Motivation and framework: RaR uses checklist-style rubrics for multi-criteria supervision in on-policy reinforcement learning, addressing tasks where correctness is not easily verifiable.The framework is designed for reasoning and real-world domains, where conventional RLVR lacks reliable automatic rewards.
  • Motivation and framework: Rubrics provide interpretable and automatable supervision that can be applied consistently across new rollouts, offering a scalable alternative to opaque reward modeling.This structured signal is intended to combine the simplicity of verifiable rewards with the expressiveness of preference-based supervision.
  • Novelty: Unlike prior rubric use for evaluation [2] [30] or concurrent preference-tuning and safety methods [7] [8] [33], RaR converts rubrics into reward functions for on-policy RL.The framework closes the rubric-to-learning loop for expert reasoning and applied real-world domains.
  • Data and implementation: RaR synthesizes instance-specific rubrics for medicine and science and releases the corresponding RaR-Medicine 1 and RaR-Science 2 training sets.The rubrics are described as prompt-specific and self-contained, with reference answers serving as proxies for expert supervision.
  • Claimed outcomes: RaR-trained models consistently outperform strong baselines and provide a stable, generalizable training signal across rubric-guided evaluations and tasks with verifiable answers.This positions rubric-based rewards as a bridge between verifiable RLVR and less structured real-world reasoning.

2. Rubrics as Rewards

RaR defines rubric-based rewards for responses evaluated against prompt-specific criteria, extending RLVR to settings where correctness is multidimensional or not strictly verifiable. It compares explicit criterion-level aggregation with implicit holistic aggregation by an LLM judge.

  • 2.1 Problem Formulation: RaR assigns each prompt k rubric items, pairing criterion weights w_j with binary functions that test whether a sampled response satisfies each criterion.The framework evaluates responses against instance-specific rubric criteria rather than relying on a single ground-truth answer.
  • 2.2 Reward Aggregation Strategies: The method investigates two complementary strategies for converting rubric feedback into scalar rewards: explicit aggregation and implicit aggregation.These strategies differ in whether the aggregation is performed manually from criterion evaluations or delegated to the judge model.
  • 2.2 Reward Aggregation Strategies: Explicit aggregation independently evaluates each criterion with an LLM judge and computes a normalized final reward.Normalization makes rewards comparable across prompts with different rubric counts or criterion weights.
  • 2.2 Reward Aggregation Strategies: Implicit aggregation passes all rubric criteria and categorical weights to an LLM judge, which directly produces a single holistic scalar reward.This approach avoids manually tuning rubric weights by delegating aggregation to the judge model.
  • 2.3 Generalization of RLVR with Rubrics as Rewards: Rubric-based reinforcement learning extends RLVR by supporting multi-dimensional, prompt-specific evaluation criteria.The formulation preserves RLVR as a special case while broadening the reward signal beyond directly verifiable correctness.
  • 2.3 Generalization of RLVR with Rubrics as Rewards: RLVR is recovered when a rubric has one criterion with unit weight whose correctness function checks the output against a known answer.This includes verifiable conditions such as exact match or test-case execution.
  • 2.3 Generalization of RLVR with Rubrics as Rewards: Rubric rewards enable flexible criterion weighting and combine objective and subjective quality dimensions when correctness is multifaceted or not strictly verifiable.Thus, RLVR is a restricted single-criterion instance of rubric-guided reinforcement learning.

3. Rubric Generation

Section 3 defines four desiderata for effective rubrics and applies them to generate scalable, instance-specific supervision for medical and science reasoning tasks. The resulting weighted rubric items support explicit and implicit reward aggregation for policy training.

  • 3.1 Desiderata: The framework identifies four desiderata for generating effective rubrics that provide human-interpretable supervision.
  • 3.1 Desiderata: Rubrics should be grounded in expert guidance, capturing the essential facts, reasoning steps, and conclusions required for correctness.Human experts or high-quality proxies are the preferred source of this grounding.
  • 3.1 Desiderata: Rubrics should comprehensively assess factual accuracy, logical coherence, completeness, style, and safety, including negative criteria for frequent or high-risk errors.
  • 3.1 Desiderata: Rubrics should weight criteria by importance so factual correctness outweighs secondary qualities such as stylistic clarity.Weights may use categorical tags, explicit numeric values, or learned schemes.
  • 3.1 Desiderata: Each rubric item should be self-contained and independently actionable for human annotators or automated judges without external context or specialized knowledge.
  • 3.2 Rubrics Creation: For medical and science reasoning datasets, LLMs generate instance-specific rubrics from golden reference answers, enabling structured-reward research without costly human annotation.This approach addresses the scarcity of human-annotated rubric datasets in these domains.
  • 3.2 Rubrics Creation: Each prompt receives a rubric of 7–20 self-contained items with numeric and categorical importance weights, using Essential, Important, Optional, and Pitfall labels in experiments.The rubrics serve directly as reward functions through explicit aggregation and related reward mechanisms.
  • 3.2 Rubrics Creation: Using OpenAI’s o3-mini and GPT-4o [14] [15] with reference-answer conditioning, the authors release RaR-Medicine and RaR-Science for GRPO training with explicit and implicit aggregation.

4. Experiments

Experiments test rubric-based rewards across medical and scientific reasoning using on-policy GRPO from Qwen2.5-7B. Evaluation covers rubric-based clinical assessment, multiple-choice science tasks, and judge alignment with human preferences.

  • Experimental Domains: RaR is evaluated across medicine and science using approximately 20k-prompt datasets with instance-specific rubrics.RaR-Medicine draws from diverse medical reasoning sources, while RaR-Science covers GPQA-Diamond-aligned scientific reasoning categories.
  • Training Setup: All models use on-policy GRPO with Qwen2.5-7B, batch size 96, learning rate 5 × 10−6, and 10% linear warmup.Training runs use one compute node with 8 NVIDIA H100 GPUs.
  • Training Pipeline: Each prompt produces k = 16 sampled responses, which gpt-4o-mini scores before GRPO updates the policy.Responses use context length 3584 and temperature 1.0; experiments compare multiple rubric reward-computation and aggregation strategies.
  • Evaluation: Evaluation spans HealthBench rubric scores, repeated greedy-decoded multiple-choice accuracy, and pairwise alignment between LLM judges and practitioner preferences.HealthBench uses 5,000 clinical conversations; multiple-choice accuracy averages 10 runs, while alignment uses roughly 3,000 paired HealthBench prompts.

5. Results

RaR delivers strong gains across rubric-based HealthBench and multiple-choice GPQA-Diamond, with RaR-Implicit the strongest overall variant. Rubric-guided rewards also improve human-preference alignment, especially for smaller judges, when rubrics incorporate expert-grounded reference answers.

  • Overall results: 31% on HealthBench and 7% on GPQA are the reported relative gains of RaR-Implicit over Direct-Likert, while rubric-guided variants outperform base and instruction-tuned policies.Gains on GPQA-Diamond indicate that rubric-induced skills generalize beyond rubric-based evaluation.
  • Overall results: RaR-Implicit also achieves small but consistent gains over Reference-Likert despite rubric quality depending on reference-answer quality.Converting open-ended answers into explicit criteria provides effective, well-aligned reward signals.
  • Overall results: RaR-Implicit is strongest overall, whereas RaR-Explicit offers more control and interpretability but can be brittle because its fixed weights are difficult to tune.The paper treats weighting as application-dependent and suggests learned or dynamic weighting as future work.
  • Alignment with human preferences: Rubric-guided scoring improves alignment with human preferences across judge sizes, with the rubric structure particularly benefiting smaller models.Synthetic rubrics without expert grounding outperform Direct-Likert but remain below expert-grounded rubrics.
  • Expert guidance for rubric generation: Reference-answer access materially improves synthetic-rubric evaluation, while human-authored and reference-grounded synthetic rubrics achieve comparable HealthBench performance.Synthetic rubrics generated without reference answers perform notably worse than the grounded alternatives.

6. Ablations

Ablations show that structured, instance-specific rubrics improve HealthBench performance, with reference-guided generation producing the strongest training signal. Rubric breadth matters more than weights or pitfall criteria, while generator capability and alignment affect reference-free rubric quality.

  • Impact of Rubric Generation Strategies: Every instance-specific rubric method outperforms rubric-free baselines on HealthBench-1k, and even the weakest RaR variant surpasses Reference-Likert.Structured rubrics provide finer-grained rewards for subjective, open-ended healthcare answers whose correctness is not binary.
  • Impact of Rubric Generation Strategies: Reference-guided synthetic rubrics consistently outperform reference-free rubrics, showing that expert signal is crucial for aligned training in high-stakes domains.Purely synthetic rubrics remain scalable but miss subtle evaluation criteria required for robust training.
  • Elements of Rubric Design: Broader rubric criteria improve HealthBench-1k performance, whereas adding rubric weights or pitfall criteria produces minimal differences.The ablation uses Qwen2.5-7B and reference-guided o3-mini rubrics on HealthBench-1k.
  • Impact of LLM Expertise on Rubric Quality: GPT-4o generates the strongest reference-free rubrics, but all reference-free models trail reference-guided rubrics; instruction tuning and reasoning ability also matter.Smaller aligned models such as GPT-4o-mini and O3-mini remain competitive with larger open-weight models.

7. Related Work

Prior work extends RLVR across domains while exposing challenges from sparse signals, verifier reliability, and limited coverage. Rubrics improve evaluation and preference optimization in difficult-to-verify settings, complementing RLHF, programmatic verification, and denser process-level feedback.

  • RLVR across domains: RLVR has expanded beyond math and code to physics, finance, policy, medicine, chemistry, psychology, economics, and healthcare [23] [31].GENERAL-REASONER reports a ten-point MMLU-Pro gain after GRPO fine-tuning, while a follow-up uses one cross-domain reward model across four fields without task-specific tweaks [23] [31].
  • Rubrics for evaluation and training: Task-specific rubrics increasingly support LLM evaluation and training in difficult-to-verify domains [2] [8] [13] [26] [27].Rubric-prompted graders outperform question-agnostic checklists in accuracy and consistency, HEALTHBENCH uses 48k clinician-written criteria with GPT-4 judges, and CPT uses rubrics to condition DPO preference pairs [2] [8] [26].
  • Learning from feedback signals: RLHF relies on many human comparisons but can introduce subjectivity and reward hacking, whereas RLVR uses programmatic checks despite potentially sparse signals [25].Programmatic verification ranges from exact-match checks on GSM8K and MATH to mixed-domain verifiers in GENERAL-REASONER and CROSS-DOMAIN RLVR [23] [31].
  • Learning from feedback signals: Process supervision provides denser step-level guidance, alongside MCTS-generated annotations and generative reward models.These approaches are presented as alternatives or complements to sparse outcome-level feedback.

8. Conclusion

The conclusion presents Rubrics as Rewards (RaR) as a post-training framework that uses structured, checklist-style rubrics as reward signals. It emphasizes rubric-guided training’s strong cross-domain performance and interpretable, modular evaluation approach.

  • Conclusion: RaR uses structured, checklist-style rubrics as reward signals for post-training language models.The framework decomposes response evaluation into transparent, multi-criteria objectives.
  • Conclusion: RaR supports both subjective and objective evaluation through transparent, multi-criteria objectives.This design is presented as a modular and interpretable alternative to preference-based methods.
  • Conclusion: Experiments show that rubric-guided training achieves strong performance across domains and significantly outperforms Likert-based baselines.

9. Limitations and Future Work

The study focuses on medicine and science to enable controlled experiments, leaving broader validation across dialogue, tool use, and other agentic tasks for future work. It also evaluates only implicit and explicit reward aggregation strategies, motivating more advanced combinations of rubric criteria.

  • Validation is limited to medicine and science, so broader testing across dialogue, tool use, and other agentic tasks remains future work.The domain choice enables controlled experiments.
  • The study evaluates only implicit and explicit reward aggregation strategies, motivating future methods that combine rubric criteria more flexibly.These strategies represent complementary extremes of flexibility and control; learning continuous criterion weights is one proposed direction.

A. Appendix … A.9 Perturbed Dataset Generation

The appendix documents RaR’s datasets, training and evaluation procedures, rubric designs, judge prompts, synthetic preference construction, and response-perturbation methodology. It also reports that rubric-guided evaluation improves policy strength and consistency across judge sizes.

  • A.1 Details of RaR-Medicine dataset: The appendix provides aggregate statistics and categorical distributions for RaR-Medicine and RaR-Science datasets, including rubric types and topic distributions.These dataset descriptions are accompanied by illustrative examples and tables covering medical and STEM training and validation data.
  • A.2 Details of RaR-Science dataset: RaR-Science examples pair questions with reference answers and weighted rubrics covering solubility direction, polarity, Ksp context, emulsifiers, chemical properties, and avoidance of ionic assumptions.The boric-acid example requires identifying greater solubility in ethanol than benzene while explaining relevant chemical principles and potential solvation effects.
  • A.3 Training Details: Rubric-guided evaluation consistently improves policies across judge sizes, with Qwen-7B-Instruct gaining +0.047 and rubric scores clustering at 0.250–0.279 versus 0.220–0.254 for Likert-only judges.The structured rubric approach helps smaller judges approximate higher-quality supervision by decomposing evaluation into interpretable binary criteria, reducing scale-related limitations.
  • A.3 Training Details: Training uses GRPO hyperparameters for Medical and Science domains, while evaluation prompts judge multiple-choice responses by whether the final option letter or answer text matches ground truth.The evaluation prompt requires only “Yes” or “No,” without rationale, for correctness judgments.
  • A.7 Synthetic Preference Set Generation: Synthetic preference data uses HEALTHBENCH [2] prompts and clinician-vetted ideal completions, generating plausible degraded counterparts through structured perturbation instructions.The perturbation process targets accuracy, completeness, clarity, safety, specificity, structure, and tone while tracking added or removed text.
  • A.8 Judge Quality impacts on Post-training: Synthetic rubric-generation prompts select 7–20 self-contained criteria with titles, category-prefixed descriptions, and weights spanning Essential, Important, Optional, and Pitfall categories.Medical prompts emphasize safety and patient-centered qualities, while science prompts cover factual correctness, reasoning, clarity, completeness, common pitfalls, and explicit answer formatting.
  • A.9 Perturbed Dataset Generation: Health-response perturbations preserve plausibility while introducing subtle degradations such as outdated information, omitted safety warnings, vague guidance, confusing structure, or reduced empathy.The procedure explicitly avoids obviously nonsensical or harmful outputs and records the text changes made.
Loading 2507.17746v2…