Source-linked AI summary
OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment
Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, Haoyu Wang
TL;DR
Existing reward models struggle to represent multifaceted human preferences, while reliable and scalable rubric construction remains difficult. OpenRubrics introduces a large rubric dataset, contrastive generation of hard rules and principles, and preference-label consistency filtering; Rubric-RM improves reward-modeling performance across benchmarks by 8.4% and transfers gains to instruction-following and biomedical policy evaluation.
Problem
Scalar or pairwise reward judgments inadequately capture multifaceted preferences, and producing reliable, scalable rubrics remains challenging.
Method
OpenRubrics uses Contrastive Rubric Generation to derive hard rules and principles from preferred and rejected responses, then filters rubrics using preference-label consistency.
Results
8.4%: Rubric-RM improves over strong baselines across eight reward-modeling benchmarks and transfers gains to instruction-following and biomedical policy evaluation.
Takeaways & Limitations
Rubrics-as-rewards provide interpretable and discriminative signals for reward modeling and support gains in policy optimization across diverse benchmarks.
Takeaways & Limitations
The framework relies on preference data and inherited model judgments, so rubrics may retain underlying biases; it is also evaluated primarily in offline preference optimization.
Abstract
from arXiv · showhide
Reward modeling lies at the core of reinforcement learning from human feedback (RLHF), yet most existing reward models rely on scalar or pairwise judgments that fail to capture the multifaceted nature of human preferences. Recent studies have explored rubrics-as-rewards (RaR) that uses structured criteria to capture multiple dimensions of response quality. However, producing rubrics that are both reliable and scalable remains a key challenge. In this work, we introduce OpenRubrics, a diverse, large-scale collection of (prompt, rubric) pairs for training rubric-generation and rubric-based reward models. To elicit discriminative and comprehensive evaluation signals, we introduce Contrastive Rubric Generation (CRG), which derives both hard rules (explicit constraints) and principles (implicit qualities) by contrasting preferred and rejected responses. We further remove noisy rubrics via preserving preference-label consistency. Across multiple reward-modeling benchmarks, our rubric-based reward model, Rubric-RM, surpasses strong size-matched baselines by 8.4%. These gains transfer to policy models on instruction-following and biomedical benchmarks.
1. Introduction
Reward modeling supplies RLHF with optimization signals, but scalar and pairwise judgments are limited for subjective, multifaceted quality. OpenRubrics addresses scalable rubric construction with complementary rubric types, contrastive generation, and consistency filtering.
- Reward modeling assigns scores or preference labels that steer language models toward helpful and harmless responses.
- Rubrics-as-rewards decompose subjective response quality into interpretable, measurable dimensions and support optimization guided by explicit principles.
- Expert-curated rubrics are costly to scale, while directly prompting language models offers limited quality control and can be expensive.
- OpenRubrics separates rubrics into hard rules for explicit constraints and principles for implicit, generalizable qualities of strong responses.
- Contrastive Rubric Generation conditions on prompts paired with chosen and rejected responses to produce comprehensive and discriminative rubrics.
- Rubric-RM outperforms strong baselines by 8.4% across eight benchmark datasets and improves policy performance on instruction-following and medical benchmarks.
2. Related Works
Prior work spans scalar, generative, pairwise, and reinforcement-learned reward models, while rubrics add structured interpretability to evaluation and alignment. OpenRubrics targets the scalability limits of frontier-model prompting.
- Standard reward models assign scalar scores using ranking losses, while newer models add synthesized reasoning, pairwise comparison, or reinforcement learning.
- OpenRubrics complements existing reward-modeling approaches with structured rubric signals and an additional layer of interpretability.
- Rubrics have been used for structured evaluation, instruction following, domain adaptation, safety, and combinations with verifiable rewards.
- Most existing rubric approaches prompt frontier language models to generate rubrics, limiting scalability.
3. Preliminaries
The paper defines rubrics as prompt-specific collections of criteria and uses them to guide pairwise reward prediction. Its framework separates rubric generation from rubric-conditioned preference modeling.
- A rubric is a prompt-specific collection of criteria, with each criterion describing an aspect of response quality.
- The rubric criteria can cover factual correctness, reasoning soundness, and style.
- Rubric-based reward modeling compares the relative quality of two candidate responses for a given prompt.
- The reward model integrates rubric criteria when producing a binary preference judgment.
- OpenRubrics trains one model to synthesize rubrics and another rubric-guided model to produce reliable, interpretable pairwise judgments.
4. OpenRubrics
OpenRubrics constructs a diverse rubric dataset from preference and instruction-tuning data, using contrastive generation to capture explicit constraints and higher-level response qualities. Preference-label consistency filtering retains rubrics that support reliable pairwise judgments, which then train rubric generation and Rubric-RM.
- Data Sources: OpenRubrics integrates publicly available preference and instruction-tuning datasets spanning general and domain-specific resources.Sources include UltraFeedback, Magpie, Skywork-Preference, and Synthetic-IF.
- Rubric Synthesis: Contrastive Rubric Generation derives hard rules for explicit prompt requirements and principles for higher-level qualities such as reasoning soundness, factuality, and style.The generator contrasts ordered candidate responses to produce task-sensitive, preference-aligned criteria.
- Rubric Filtering: Generated rubrics are retained only when group-level verification reaches τ = 0.5 and pairwise predictions match the human preference labels.This preference-label consistency step removes rubrics that do not faithfully capture the preference signal.
- Model Training: The retained pairs form a rubric-conditioned preference dataset used to fine-tune a rubric generator and train Rubric-RM.Rubric-RM predicts preference labels from the prompt, response pair, and generated rubric.
- Rubric Statistics: Rubric statistics show coverage across instruction following, reasoning, and general helpfulness, with semantic prompt diversity visualized using t-SNE.These statistics are presented as evidence that the synthesized rubrics provide comprehensive and discriminative coverage.
- Inference: At inference, Rubric-RM first generates or retrieves a rubric for the instruction, then predicts which of two responses is better using that rubric.The two-stage process explicitly grounds the final judgment in rubric criteria.
5. Experiment
Rubric-RM is evaluated as a reward model and as a policy-optimization signal across reward-modeling, instruction-following, alignment, biomedical, and efficiency benchmarks. It consistently improves over comparable baselines, with gains strengthened by voting and domain-specific rubric training.
- Experiment Settings: Rubric-RM trains a rubric generator and judge from contrastively produced rubrics filtered by preference–label consistency.The default 8B variant fine-tunes both components from Qwen-3-8B, while voting@5 aggregates five independently sampled judge trajectories.
- Experiment Settings: Rubric-RM is compared with same-scale white-box reward models, larger reasoning models, API judges, and a directly prompted Qwen-3-8B rubric-and-judge baseline.Evaluation covers RewardBench, RM-Bench, PPE-IFEval, FollowBench, InfoBench, IFBench, RewardBench2, and HealthBench.
- Reward Modeling: 73.0 is the best overall average for Rubric-RM-8B-voting@5, exceeding RM-R1-14B at 71.7 and Rubric+Judge API at 71.3.Rubric-RM-4B scores 68.4 and Rubric-RM-8B scores 70.1, while voting reaches 70.3 and 73.0, respectively.
- Reward Modeling: 81.5 on FollowBench and 83.8 on InfoBench show strong reward-model performance on fine-grained instruction adherence.The results are presented as substantially higher than baselines such as JudgeLRM and RRM.
- Policy Optimization: 79.5 on IFEval and 83.0 on InfoBench are achieved by policies trained with Rubric-RM, versus 76.0 on IFEval with Skywork/ArmoRM.Rubric-RM also reaches 30.3 on IFBench, compared with 28.2 for RLCF and 22.4–22.8 for RLMT-based methods.
- Alignment and Biomedical Evaluation: 48.6 is Rubric-RM’s best overall average among open-source reward models under DPO, while WildBench reaches 53.6 versus 48.7 for the base model.On HealthBench, Rubric-RM scores 68.3 and voting@5 reaches 72.9; DPO with Rubric-RM reaches 23.8 from a 21.6 starting model.
- Efficiency: 130.77s on 100 RewardBench2 prompts is no slower than the reported 7B reasoning-model baselines despite using separate rubric-generator and judge components.The paper attributes efficiency to short rubric generation rather than long Chain-of-Thought decoding.
6. Conclusion
OpenRubrics provides a scalable framework for generating high-quality rubric signals and training rubric-based reward models. Rubric-RM improves reward-model performance across benchmarks and transfers gains to policy optimization.
- OpenRubrics constructs a large-scale dataset and framework for scalable, high-quality rubric generation.The framework targets interpretable and discriminative rubric signals for alignment.
- Contrastive Rubric Generation decomposes evaluation into hard rules and principles, while preference–label consistency filtering removes noisy rubric signals.This combines explicit constraints with implicit qualities and filters rubrics for consistency with preferences.
- Rubric-RM delivers an average 8.4% improvement across benchmarks and further boosts policy performance on diverse benchmarks with offline reinforcement learning.The reported gains extend beyond reward modeling to policy optimization.
- The results position rubrics-as-rewards as a practical foundation for transparent and generalizable LLM alignment.
Limitations
OpenRubrics has limitations related to inherited biases, pairwise evaluation, and reliance on offline preference optimization. Extending the framework to broader evaluation and training settings remains open.
- Rubrics may reflect biases in underlying models and datasets, especially for subjective or culturally nuanced criteria.Preference–label consistency filtering reduces noise but does not eliminate these inherited biases.
- The framework focuses on pairwise comparative evaluation rather than absolute scoring or multi-response ranking.Extending rubric-based rewards to those settings remains an open challenge.
- The evaluation primarily covers offline preference optimization, leaving interactions with online RLHF, exploration, and long-horizon policy learning unresolved.
B.1. Decoding and efficiency protocol
The evaluation protocol matches decoding budgets across models and uses a unified execution stack for throughput comparisons. Rubrics can be cached and reused to amortize judging costs.
- All models use matched temperature, maximum-token, and stop-condition decoding budgets according to benchmark recommendations.
- The unified vLLM execution stack supports throughput-fair comparisons across models.
- Rubrics generated in stage (i) are cacheable and reusable across examples, amortizing costs in large-scale judging and preference optimization.
B.2. Hyper-parameters
The appendix specifies training and sampling hyper-parameters for Rubric-RM, policy models, OpenRubrics curation, and inference. Baseline sampling parameters follow the corresponding official implementations and papers.
- Rubric-RM and policy-model training use hyper-parameters detailed in Table 7 and conducted in LLaMA-Factory.
- OpenRubrics curation and Rubric-RM inference use sampling parameters presented in Table 8.
- Baseline methods use sampling parameters from their official implementations and papers.
B.3. Additional Case Study
The case study shows that rubric-aware evaluation can resist citation hallucinations by explicitly encoding requirements such as recency and verifiability. The accompanying prompts operationalize rubric construction through abstraction, category separation, and validation checks.
- Additional Case Study: Baselines incorrectly favored the better response after hallucinating that it lacked a date or citation.The cited response actually included a BloombergNEF quote, a May 16, 2024 publication date, and concrete figures.
- Additional Case Study: Rubric-RM identified recency and verifiability as hard requirements and favored the response satisfying them.Its rubric also required a quote, date, concise summary, and economic implications.
- Prompt Methodology: The rubric-generation process extracts explicit prompt constraints as non-negotiable hard rules.These rules are labeled [Hard Rule] in the final rubric.
- Prompt Methodology: It analyzes preference differences, then abstracts topic-specific observations into universal principles applicable across domains.Concrete facts, names, events, topics, and response indices are prohibited in final principles.
- Prompt Methodology: Final rubrics separate [Hard Rule] and [Principle] criteria while requiring complete coverage, distinctness, concise wording, and consistent third-person formatting.A pre-output validation check verifies abstraction, inclusion of hard rules, non-overlap, and stylistic consistency.