Source-linked AI summary
Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
William F. Shen, Xinchi Qiu, Chenxi Whitehouse, Lisa Alazraki, Shashwat Goel, Francesco Barbieri, Timon Willi, Akhil Mathur, Ilias Leontiadis
TL;DR
Rubric-based LLM judging is brittle because existing rubric sets can lack coverage and contain noisy, misaligned, overlapping, or correlated criteria. RRD recursively decomposes and filters rubrics, then weights them for correlation, improving judge accuracy and reward-model effectiveness for RFT, with gains transferring to downstream policy evaluations.
Problem
Existing rubric generation can miss diverse quality dimensions and introduce misaligned, overlapping, or highly correlated criteria, weakening preference alignment and limiting gains from rubric-based judging and RFT.
Method
RRD recursively decomposes high-level criteria into granular discriminative rubrics, filters unreliable and redundant rubrics, and uses correlation-aware weighting to stabilize aggregation.
Results
RRD achieves top performance across judge settings, with up to 17.7% improvement on JudgeBench for GPT-4o, and RRDWU reaches 82.8% and 71.1% on BiGGen Bench for Qwen3-4B and Llama3.1-8B policies, respectively.
Takeaways & Limitations
RRD provides informative, comprehensive, and non-redundant rubrics that improve training-free LLM judging and serve as high-fidelity reward models for RFT.
Takeaways & Limitations
The theoretical grounding assumes every rubric has positive edge and that rubric noise is bounded with pairwise correlations below one.
Abstract
from arXiv · showhide
Recently, rubrics have been used to guide LLM judges in capturing subjective, nuanced, multi-dimensional human preferences, and have been extended from evaluation to reward signals for reinforcement fine-tuning (RFT). However, rubric generation remains hard to control: rubrics often lack coverage, conflate dimensions, misalign preference direction, and contain redundant or highly correlated criteria, degrading judge accuracy and producing suboptimal rewards during RFT. We propose RRD, a principled framework for rubric refinement built on a recursive decompose-filter cycle. RRD decomposes coarse rubrics into fine-grained, discriminative criteria, expanding coverage while sharpening separation between responses. A complementary filtering mechanism removes misaligned and redundant rubrics, and a correlation-aware weighting scheme prevents over-representing highly correlated criteria, yielding rubric sets that are informative, comprehensive, and non-redundant. Empirically, RRD delivers large, consistent gains across both evaluation and training: it improves preference-judgment accuracy on JudgeBench and PPE for both GPT-4o and Llama3.1-405B judges, achieving top performance in all settings with up to +17.7 points on JudgeBench. When used as the reward source for RFT on WildChat, it yields substantially stronger and more stable learning signals, boosting reward by up to 160% (Qwen3-4B) and 60% (Llama3.1-8B) versus 10-20% for prior rubric baselines, with gains that transfer to HealthBench-Hard and BiGGen Bench. Overall, RRD establishes recursive rubric refinement as a scalable and interpretable foundation for LLM judging and reward modeling in open-ended domains.
1 Introduction
LLM judging of open-ended tasks remains brittle because quality is subjective and multi-attribute, while rubric generation can lack coverage and introduce noisy, redundant signals. RRD recursively decomposes and filters rubrics, improving judgment accuracy and producing stronger reward signals for RFT.
- Subjective, multi-attribute quality makes LLM judges vulnerable to bias, inconsistency, and limited transparency.
- Preference supervision for open-ended reinforcement learning must incorporate diverse criteria matching generation complexity.
- Naively generated rubrics reduce GPT-4o JudgeBench accuracy from 55.6% to 42.9%, 13 points below using no rubrics.
- RRD decomposes broad criteria into discriminative subpoints, filters misaligned or conflicting rubrics, and down-weights correlated ones.These components aim to expand coverage while producing more stable and informative assessments.
- 17.7% improvement on JudgeBench for GPT-4o is achieved by RRD WU, which tops all evaluated judgment settings.
- 160% reward improvement for Qwen3-4B and 60% for Llama3.1-8B exceed the approximately 10–20% gains of prior rubric baselines.The resulting policies also improve on BiGGen Bench and HealthBench-Hard for both model families.
2 RRD Framework
RRD frames rubric quality as a balance of informative, comprehensive, and non-redundant criteria, then operationalizes this through recursive decomposition, filtering, and correlation-aware weighting. Its procedure turns broad criteria into discriminative subdimensions, removes misaligned or overlapping rubrics, and assigns weights that reduce correlated over-representation.
- Rubric-based judge formulation: Rubric-based judges evaluate responses through separately assessed predicates aggregated with nonnegative weights into a rubric reward.
- Rubric quality desiderata: Effective rubric sets should be informative, comprehensive, and non-redundant because rubric quality shapes judgment accuracy and RFT reward fidelity.
- Theoretical grounding: The theoretical objective maximizes weighted preference edge relative to weighted residual variance, motivating decomposition, positivity, non-redundancy, and weight optimization.
- RRD framework: RRD decomposes broad rubrics into finer dimensions, filters misaligned and redundant criteria, and optimizes weights to reduce correlation-driven over-representation.
- Recursive refinement: RRD proposes initial rubrics from task prompts and sample responses, recursively refines criteria applying broadly, and stops after rejected proposals exceed a tunable threshold.
- Correlation-aware weighting: Whitened weighting estimates rubric covariance from unlabeled data, removes correlations through Σ^-1/2, and applies equal weighting in whitened coordinates.
3 RRD-based LLM Judge Results
RRD improves open-form preference judging across JudgeBench and PPE for both GPT-4o and Llama3.1-405B, with whitened-uniform weighting performing best. Its recursive process expands rubric sets adaptively, while ablations show robustness to deeper exploration and benefits from strong, diverse sample-response generators.
- Benchmark results: RRD variants consistently outperform rubric-generation baselines across JudgeBench and PPE for GPT-4o and Llama3.1-405B judges.The evaluation uses JudgeBench and PPE, including 350 GPT-4o-generated preference pairs and 10.2K human preference pairs from Chatbot Arena.
- Benchmark results: +17.7 points: RRDWU raises GPT-4o JudgeBench accuracy from 55.6% to 73.3%, while Llama-3.1-405B gains 6.6 points.
- Weighting comparison: Whitened weighting outperforms uniform and LLM-assigned weighting, supporting correlation-aware aggregation for open-ended tasks with multiple non-dominating dimensions.
- Rubric-count dynamics: 7.4 to approximately 20 rubrics: recursive expansion plateaus by iteration 3, while increasing count variance adapts evaluation depth to task complexity.
- Termination-threshold ablation: RRDWU remains accurate at termination threshold 20, whereas lower thresholds underperform and deeper exploration can add diminishing returns or correlated noise to basic variants.
- Sample-response ablation: GPT-4o plus Gemini 2.5-Pro samples perform best, and diverse frontier-model outputs improve rubric generation relative to single-model or weaker-model mixtures.
4 RRD-based RFT
The RFT experiments test whether RRD’s rubric-based judging improvements produce stronger learning signals and better downstream policies. Across training and evaluation, RRD improves reward dynamics and policy performance on both general and high-stakes open-ended tasks.
- Dataset and Training: RFT evaluates reward reliability during training and downstream policy performance on in-domain and out-of-domain tasks.The study trains on 4K English, non-toxic, de-duplicated WildChat prompts and evaluates policies on BiGGen Bench and HealthBench-Hard.
- Reward Dynamics during Training: RRD rewards rise rapidly during the first ∼50–100 steps and continue improving, while prior baselines plateau early at low gains.This pattern appears for both Qwen3-4B and Llama3.1-8B-Instruct.
- Reward Dynamics during Training: RRDWU reaches roughly 150–160% reward improvement on Qwen3-4B and about 55–60% on Llama3.1-8B-Instruct, versus ∼10–20% for baselines.RRDWU also shows the smoothest and most stable curves, while both RRD variants outperform alternatives.
- Policy Performance on BiGGen Bench: RRDWU achieves the highest overall BiGGen Bench scores for Qwen3-4B and Llama3.1-8B, reaching 82.8% and 71.1%, respectively.Its strongest capability improvements are in instruction following, refinement, and reasoning, while safety performance remains robust.
- Generalization to High-Stakes Domains: On HealthBench-Hard with Qwen3-4B, RRDWU improves instruction following, accuracy, and completeness by +16.0%, +5.5%, and +12.5% points, respectively.The results indicate transfer to physician-authored, high-stakes clinical evaluation criteria.
5 Related Works
Prior work has moved from holistic LLM judges toward rubric-assisted evaluation and from verifiable rewards toward LLM-judge-based rewards for open-ended tasks. These approaches address important gaps but remain limited by bias, inconsistency, opacity, heuristic rubrics, and the lack of ground-truth verifiers.
- LLM-as-a-Judge: Holistic LLM judges can correlate with human preferences but suffer from bias, inconsistency, and opacity on subjective, nuanced, multidimensional tasks.Rubric-assisted judges were introduced to address these limitations, though many use static or heuristic rubrics.
- Rubrics-based Rewards: Rubric-based rewards extend structured evaluation signals into reinforcement fine-tuning for complex open-ended tasks.Prior methods include Propose–Evaluate–Revise, score-differential optimization, and domain-specific rubric generation using prompt context and exemplars.
- Reinforcement Fine-Tuning (RFT) for Open-Ended Tasks: Extending RLVR to open-ended tasks remains difficult because these tasks lack ground-truth verifiers.Recent work therefore uses LLM judges as proxy reward models for algorithms such as GRPO.
6 Conclusion
RRD generates informative, comprehensive, and non-redundant rubrics for LLM judging and reward modeling in open-ended tasks. Its empirical results support stronger judge accuracy and higher human-preference alignment after RFT, motivating structured and statistically grounded rubric generation.
- Conclusion: RRD recursively decomposes high-level criteria into granular, discriminative rubrics, filters misaligned and redundant criteria, and weights correlated metrics to limit over-representation.The framework targets both training-free judge accuracy and reward-model quality for RFT.
- Conclusion: Policies trained with RRD-derived rewards exhibit stronger alignment with human preferences on complex, open-ended generation tasks.The conclusion presents RRD as a pathway toward scalable, interpretable, and reliable alignment.
Appendix
Algorithm 1 describes RRD as an iterative process that evaluates rubrics against sampled responses, decomposes insufficiently discriminative criteria, and filters conflicts or overlaps until enough final rubrics remain.
- Algorithm 1: RRD: Recursive Rubric Decomposition: For each rubric, the procedure collects sampled responses that satisfy it and decomposes the rubric when the satisfaction set is sufficiently large.Rubric evaluation uses the LLM-based proposer’s rubric judgments over sampled responses.
- Algorithm 1: RRD: Recursive Rubric Decomposition: RRD initializes a final rubric set from G0 and iterates until the filtered set contains N rubrics.The loop maintains an iteration counter and processes each rubric in the current set.
- Algorithm 1: RRD: Recursive Rubric Decomposition: The filter removes rubrics having a conflicting or overlapping counterpart, then returns the resulting set as the final rubric collection.This filtering step is applied after decomposition in each iteration.
B Additional Notes on Derivations and Proofs
The derivations formalize how rubric-score errors depend on rubric information, correlation, and covariance estimation. Under stated assumptions, expanding positively informative rubrics helps, while correlation-aware whitening yields stable weights.
- Probability bound: Under symmetric assumptions, adding uncorrelated rubrics with positive information edges consistently reduces prediction error.The information edge is positive when a rubric’s prediction direction aligns with human preference.
- Correlation-aware weighting: When rubric information edges are unobservable and criteria have comparable importance, aggregation must account explicitly for rubric correlations.This motivates correlation-aware normalization through an effective weighting scheme.
- Covariance estimation: The covariance estimator is analyzed using concentration bounds based on sub-Gaussian rubric scores and matrix Bernstein inequalities.The analysis controls operator-norm fluctuations and supports stable estimation of the covariance structure.
- Covariance estimation: Whitened uniform weights computed from the sample covariance are consistent and numerically stable.The proof uses spectral perturbation control for inverse square roots of positive-definite covariance matrices.
- Whitened-space weighting: The theoretical weighting objective selects weights proportional to Σ^-1/2 1 under exchangeable positive-orthant edge assumptions.In whitened coordinates, the least-favorable direction is handled by choosing the barycenter direction, which maps back to w ∝Σ^-1/2 1.
C Discussion
The discussion treats filtering and recursive decomposition as safeguards that require domain-sensitive control. Misalignment filtering can fail on value-sensitive criteria, while aggressive decomposition can produce overly specific, unstable rubrics.
- Misalignment filtering: The misalignment filter flags rubrics that systematically prefer a weaker reference model over a stronger one.The authors frame this directionality heuristic as a conservative sanity-check rather than a normative definition of quality.
- Misalignment filtering: The filter may fail when humans prefer caution, brevity, calibrated uncertainty, or particular refusal behaviors.The authors recommend disabling it in such domains or replacing it with multi-reference checks or axis-specific calibration.
- Controlling over-decomposition: Aggressive decomposition can fragment criteria into overly specific sub-rubrics that track incidental response artifacts instead of stable preference dimensions.A small coverage threshold is given as an example of an aggressive trigger.
- Controlling over-decomposition: Non-redundancy filtering removes duplicative, conflicting, or near-paraphrase candidates during recursive refinement.This safeguard limits rubric growth while preserving genuinely discriminative sub-dimensions.
- Controlling over-decomposition: A rejection-based early-stopping threshold halts recursion when a round produces too many invalid or non-novel candidates.Together with filtering, this bounds rubric-set growth and reduces degeneration into sample-specific criteria.
D RFT Training Details
The RFT setup optimizes a policy using group-based policy gradients and applies Dr.GRPO consistently across baselines and RRD variants. Dr.GRPO removes normalization terms that otherwise induce optimization bias.
- RFT objective: RFT aligns policy πθ by maximizing expected return over sampled prompts and policy-generated responses.The trajectory return supplies the reward signal, with an optional reference policy πref.
- Optimization: PPO-style policy-gradient methods optimize a clipped surrogate objective using importance ratios against the pre-update policy.The ratio compares current and previous token probabilities.
- Optimization: GRPO samples a group of G responses per question and assigns group-relative advantages across response tokens.This avoids requiring a learned value function.
- Optimization: Dr.GRPO removes per-response length normalization and per-question standard-deviation normalization to mitigate optimization bias.The paper reports that this yields an unbiased policy-gradient estimator and improved token efficiency.
- Experimental setup: All experiments consistently use Dr.GRPO for RFT training across baselines and RRD variants.Shared training hyperparameters are provided in Table 2 using the verl library.
E RFT Training Results
RRDWU produces the strongest reported downstream RFT performance across BiGGen Bench and HealthBench-Hard. Improvements span multiple capabilities and remain robust across Qwen3-4B and Llama3.1-8B backbones.
- BiGGen Bench: RRDWU consistently improves BiGGen Bench performance across both Qwen3-4B and Llama3.1-8B backbones, including evaluation axes and overall scores.The table compares base pre-RFT models with models fine-tuned using multiple methods.
- HealthBench-Hard: On HealthBench-Hard, RRDWU outperforms the base model and RFT baselines on key axes including instruction following, accuracy, completeness, context, and overall score.Comparable communication performance is reported relative to other RRD variants, with similar gains using Llama3.1-8B.
F Prompts
The prompts specify how to generate, decompose, filter, and evaluate rubrics for LLM-as-judge systems. They emphasize comprehensive, atomic, objective, self-contained, verifiable, non-overlapping, and non-conflicting criteria.
- F.1 Rubric Generation: Initial rubric generation asks for prompt-specific criteria that collectively cover the most important dimensions of a correct and helpful response.Rubrics should be consistently judgeable, avoid vague wording, and use clear binary pass/fail boundaries.
- F.1 Rubric Generation: Rubric sets should be mutually exclusive and collectively exhaustive, including requested facts plus supporting justification, reasoning, and logic.The same model error should not be punished multiple times.
- F.1 Rubric Generation: Each criterion should evaluate one distinct aspect rather than bundle multiple requirements into a stacked rubric.The prompts illustrate splitting a criterion about identifying George Washington and his two terms into two separate rubrics.
- F.1 Rubric Generation: Criteria should be binary, objective, specific, self-contained, and verifiable without requiring external search.The prompts contrast vague accuracy requirements and incomplete references with explicit, internally checkable criteria.
- F.1 Rubric Decomposition: Recursive decomposition uses the prompt, candidate responses, a coarse current rubric, and non-overlapping existing rubrics to propose exactly two granular discriminative rubrics.New rubrics should preserve critical information, remain consistently judgeable, and target dimensions such as completeness, constraints, justification, edge cases, or format faithfulness.
- F.2 Filtering: Overlap filtering flags a new rubric when it has the same intent, is a strict subset or superset, or has at least 70% of its meaning covered by an existing rubric.The comparison is semantic rather than lexical and ignores trivial phrasing differences.
- F.2 Filtering: Conflict filtering flags rubrics expressing opposite polarity for the same requirement while ignoring different targets, contexts, and orthogonal dimensions.Examples include requiring versus forbidding the same content or preferring more versus less of the same property.
- F.3 Evaluation: The evaluation prompt instructs a judge to output YES when a response satisfies the rubric criterion and NO otherwise.The surrounding prompts require strict output formats for rubric generation and filtering decisions.
G Qualitative Examples
The qualitative examples show RRD transforming broad initial rubrics into more granular criteria across sales emails, simple reasoning, history, legal analysis, and skill recommendations. The refined examples add concrete content requirements, explanations, edge cases, and persuasive or creative dimensions while preserving task constraints.
- Overview: Representative outputs compare initially proposed rubrics with refined rubrics generated through recursive decomposition and filtering.The examples are intended to illustrate increased granularity and coverage qualitatively rather than display every dataset instance.
- JudgeBench Dataset: In the malicious-prosecution example, final rubrics separately address tort elements, the propensity rule, and the character-evidence exception.These criteria refine the initial focus on relevance, admissibility, legal elements, and propensity reasoning.
- PPE Preference Dataset: In the website-speed sales-email example, refinement expands broad requirements into specific revenue claims, delay-loss metrics, service details, calls to action, analogies, credibility, urgency, and competitive advantage.The initial criteria already constrain audience, benefits, length, jargon, and calls to action.
- PPE Preference Dataset: In the apple-count example, refined rubrics add creative narrative criteria while retaining the stated fact that today’s count is 10 apples.The initial rubrics focus on explaining why yesterday’s consumption is irrelevant to today’s count.
- WildChat Dataset: For the Greek-thought question, refinement introduces concrete criteria about Thales, Kosmos, Pythagorean mathematics, myth-to-logic transitions, rational order, Aristotle, and empirical methods.The initial rubrics specify historical framing, conceptual contributions, figures or schools, methodological foundations, and avoidance of anachronism.
- WildChat Dataset: The six-month skill example begins with learnability and monetization, then adds explicit skill identification, realistic earnings, plausible applications, and alignment with both constraints.The refined requirements remain tied to the six-month time limit and financial goal.