OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment

1. Introduction

• Rubrics-as-rewards (RaR) (Gunjal et al., 2025) decompose subjective response quality into interpretable criteria, addressing limits of scalar judgments in domains where correctness is not captured by binary signals.The introduction identifies costly expert curation and limited quality control or high API costs in direct-prompting approaches as barriers to scalable, reliable rubric construction.Source: §1 · pp. 1–2
• Contrastive Rubric Generation (CRG) uses chosen and rejected responses to derive discriminative principles, while preference-label consistency filtering retains rubrics that predict preferences correctly.OpenRubrics distinguishes hard rules for explicit constraints from principles capturing implicit, generalizable qualities of strong responses.Source: §1 · p. 2
• OpenRubrics is a large-scale, diverse collection of prompt–rubric pairs intended to support training rubric-generation models and rubric-informed reward models at scale.Its rubrics cover both surface-level requirements and deeper dimensions of response quality.Source: §1 · p. 2
• 8.4%: Across eight benchmark datasets, Rubric-RM consistently outperforms strong baselines and yields gains on instruction-following and medical benchmarks when integrated into policy optimization.Case studies also show that combining hard rules and principles helps reduce false positives from overly long outputs.Source: §1 · p. 2

Reward Modeling

• Prior reward models use Bradley–Terry scalar scoring, CoT-generative scoring, pairwise comparison, or reinforcement-learning optimization (Bradley and Terry, 1952; Liu et al., 2025a; Ankner et al., 2024; Yu et al., 2025; Zhang et al., 2025b; Liang et al., 2025; Xu et al., 2025a; Liu et al., 2025b; Chen et al., 2025a,b; Whitehouse et al., 2025; Guo et al., 2025b).OpenRubrics complements these approaches with structured rubric-based evaluation signals that add interpretability and yield performance gains.Source: §2 · pp. 2–3

Rubrics as Rewards

• Rubrics support generation assessment, instruction following, domain adaptation, safety, and reasoning, but frontier-LLM prompting limits scalability and consistency (Arora et al., 2025; Hashemi et al., 2024; Pathak et al., 2025; Lin et al., 2025; Viswanathan et al., 2025; Gunjal et al., 2025; Mu et al., 2024; Huang et al., 2025; Zhou et al., 2025).OpenRubrics proposes scalable, lower-cost synthetic rubric generation to improve reward quality and interpretability.Source: §2 · p. 3
• Unlike Zhang et al. (2025a), which iteratively refines rubrics to mitigate reward over-optimization, OpenRubrics emphasizes scalable synthesis and rubric–preference consistency.The paper presents these as distinct focuses within concurrent work on rubric generation.Source: §2 · p. 3

• A rubric is a prompt-specific set of criteria describing aspects of response quality, such as factual correctness, reasoning soundness, and style.For prompt x, the rubric ℛ(x) contains k criteria, each describing an aspect of response quality.Source: §3 · p. 3
• In the comparative setting, a rubric-based reward model integrates rubric criteria to predict a binary preference between two candidate responses to a prompt.Source: §3 · p. 3
• The framework builds separate datasets to train a model that generates prompt-specific rubrics and a rubric-guided reward model that makes reliable, interpretable pairwise judgments.This two-stage design decomposes evaluation into rubric generation and rubric-conditioned reward prediction.Source: §3 · p. 3

4.1. Data Construction

• The dataset combines general preference and instruction data with scientific and medical resources, constructing preference pairs from response rankings or verification outcomes.For UltraFeedback, the highest- and lowest-scoring responses form the pair; Synthetic-IF uses verification results, while MegaScience and Medical-o1 use reward-model rankings.Source: §4.1 · pp. 4–5

4.2. Rubrics Synthesis

• Rubrics combine explicit prompt requirements (hard rules) with qualitative principles, are generated by contrasting ranked responses, and are retained only when rubric-level accuracy reaches 0.5 and the current pair is predicted correctly.The approach uses induced pairwise comparisons and produces rubrics intended to distinguish higher- from lower-preference responses.Source: §4.2 · pp. 5–6
• Rubric statistics assess domain coverage, the balance of hard rules and principles, prompt and rubric length, and semantic diversity, and are reported to show comprehensive yet discriminative coverage.Prompt-topic diversity is visualized with t-SNE embeddings.Source: §4.2 · p. 6, Figure 3 · p. 6

4.3. Reward Model Training and Inference

• A prompt-conditioned rubric generator and a reward model trained on prompts, response pairs, and rubrics produce preference judgments grounded in rubric criteria.At inference, the generator creates or retrieves a rubric for the instruction, then the reward model judges the candidate responses using it.Source: §4.3 · pp. 6–7

5.1. Datasets and Experiment Settings

• Rubric-RM-8B fine-tunes a rubric generator and judge on curated OpenRubrics, using the science-related slice by default for HealthBench and medical evaluation.Rubrics derive hard rules and principles from chosen and rejected responses and are filtered for preference-label consistency; the study also reports a five-trajectory majority-vote variant.Source: §5.1 · p. 7

5.2. Performance of Rubric-RM

• Rubric-RM-8B-voting@5 achieves the best overall average of 73.0, while Rubric-RM-4B scores 68.4 versus 61.7 for the strongest 7B competitor.Rubric-RM-8B scores 70.1, and Rubric-RM-8B-voting@5 exceeds RM-R1-14B (71.7) and the Rubric+Judge API (71.3).Source: §5.2 · p. 8, Table 1 · p. 8

5.3.1. Instruction-Following Evaluation

• Rubric-RM-trained policies score 79.5 on IFEval versus 76.0 with Skywork/ArmoRM, 83.0 on InfoBench, and 30.3 on IFBench versus 28.2 with RLCF.The reported IFBench scores for RLMT-based methods are 22.4–22.8; rubric-based rewards are described as improving constrained instruction following.Source: Table 2 · p. 9, Figure 4 · p. 9, §5.3.1 · p. 10

5.3.2. General Alignment Evaluation

• With DPO, Rubric-RM reaches a 48.6 overall average among open-source reward models, scores 53.1 on style-controlled Arena-Hard and 47.0 on AlpacaEval, and yields 53.6 on WildBench.WildBench improves by +4.9 over the base model (48.7), with gains across task categories.Source: Table 3 · p. 10, Table 4 · p. 10, §5.3.2 · p. 11

5.4. Rubric-RM for BioMedical Domain

• Rubric-RM scores 68.3 on HealthBench versus 63.3 for RRM-7B, and voting@5 reaches 72.9; DPO with Rubric-RM scores 23.8 on HealthBench policy optimization.Directly prompting Qwen-3-8B with Rubric+Judge scores 51.8, while DPO with ArmoRM and RM-R1-7B scores 22.5 and 22.7, respectively.Source: §5.4 · p. 11, Figure 5 · p. 11

5.5. Efficiency Comparison

• Rubric-RM evaluates 100 prompts in 130.77s, compared with 203.4s for RRM-7B and 170.8–382.0s for RM-R1-7B/14B.Short rubric generation and lightweight judging reduce decoding, and rubrics can be cached to remove generation cost during large-scale scoring.Source: §5.5 · p. 12, Table 6 · p. 12

5.6. Case Studies

• In the RewardBench case study, Rubric-RM rejects a vivid answer that violates the fewer-than-two-paragraph constraint before comparing higher-level qualities.The baselines prefer the longer answer; Rubric-RM’s structural check prevents this constraint violation and verbosity bias.Source: §5.6 · pp. 12–13, Table 5 · p. 12

6. Conclusion

• OpenRubrics is a large-scale dataset and framework for scalable, high-quality rubric generation, decomposing evaluation into hard rules and principles through Contrastive Rubric Generation.Preference-label consistency filtering yields interpretable, discriminative rubric signals that better align with human judgment.Source: §6 · p. 13
• 8.4% average improvement across benchmarks is achieved by Rubric-RM, with further policy-performance gains on diverse benchmarks using offline reinforcement learning.The paper presents rubrics-as-rewards as a practical foundation for transparent and generalizable LLM alignment.Source: §6 · p. 13

Limitations

• Rubrics derive from preference contrasts and inherited model judgments; consistency filtering reduces noise but may not remove model and dataset biases, especially for subjective or culturally nuanced criteria.Source: Limitations · p. 13
• The framework focuses on pairwise comparisons, while absolute scoring and multi-response ranking remain open challenges.Source: Limitations · p. 13
• Evaluation focuses primarily on offline preference optimization, leaving interactions with online RLHF, exploration, and long-horizon policy learning unresolved.Source: Limitations · p. 13

A. Additional Rubric Statistics

• Figure 6 presents additional rubric statistics on the number of preference pairs.Source: A. Additional Rubric Statistics · p. 19

B.1. Decoding and efficiency protocol

• Models use matched benchmark-recommended decoding budgets and a unified vLLM execution stack for throughput-fair comparisons.Decoding budgets match on temperature, maximum tokens, and stop conditions.Source: §B.1 · p. 19
• Efficiency is measured as wall-clock time to score a fixed prompt set, with cacheable stage (i) rubrics amortizing costs at scale.The paper reports these measurements in Table 6.Source: §B.1 · p. 19

B.2. Hyper-parameters

• Table 7 reports Rubric-RM and policy-model training hyper-parameters, while Table 8 reports OpenRubrics curation and Rubric-RM inference sampling parameters.Training used LLaMA-Factory; baseline sampling parameters came from official implementations and papers.Source: §B.2 · p. 19

B.3. Additional Case Study

• Rubric-RM correctly prefers the response meeting citation, date, and stance requirements, while baselines are misled by a hallucinated missing-date claim.The case contrasts longer answers with a subtle quality gap; Rubric-RM identifies recency and verifiability as hard requirements and resists citation hallucinations and academic-looking prose.Source: §B.3 · pp. 19–20, Table 9 · p. 22

B.4. Prompts

• The listwise prompt extracts explicit requirements as hard rules, contrasts responses ordered by preference, then abstracts differences into universal principles.It requires distinct, concise criteria, a numbered third-person format labeled [Hard Rule] or [Principle], and a validation check before output; baseline prompts came from official implementations and papers.Source: §B.4 · pp. 20, 23, 24
• The pairwise prompt follows the same three-step abstraction process, deriving principles from chosen-versus-rejected differences while preserving explicit request requirements as hard rules.It bars topic-specific details in the final rubric and calls for concise, unique criteria that cover both explicit requirements and implicit quality standards.Source: §B.4 · p. 25
• The general-domain judge prompt first identifies an objective gatekeeper, evaluates each response against every rubric criterion with concrete evidence, then makes and justifies a final choice.Its required output separates the compliance check, criterion-by-criterion analysis, and final judgment.Source: §B.4 · p. 27
• The judge prompt identifies one objective Gatekeeper Criterion, evaluates each response against it and all other rubric criteria, then selects a winner with a justification.Subjective quality criteria cannot serve as Gatekeepers.Source: §B.4 · p. 28
• The judge prompt requires separate compliance, response-analysis, and final-judgment sections, including criterion labels and a named winner.The output template provides fields for each response’s hard rules and principles, final justification, and winner.Source: §B.4 · p. 28
• Rubric generation separates explicit requirements into [Hard Rule] criteria from domain-agnostic quality principles and requires coverage of critical explicit and implicit standards.Topic-specific references invalidate a rubric item.Source: §B.4 · p. 29
• The rubric-generation prompt requires distinct, precise criteria, merges overlaps, and specifies numbered third-person items labeled [Hard Rule] or [Principle] without explanations.The required output contains only the rubrics.Source: §B.4 · p. 29