PaperlayerView exact revision on arXiv

Source-linked AI summary

Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training

Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, Lifeng Jin

arXiv:2509.21500v3cs.LGcs.AI

TL;DR

RFT can over-optimize imperfect proxy rewards, especially when they misrank responses in the high-reward tail. This paper analyzes that failure, constructs rubric-based rewards from off-policy exemplars through iterative refinement, and reports effective post-training with reduced over-optimization. The approach remains limited by simple score aggregation and by the difficulty of scaling Bradley-Terry alternatives in specialized domains.

  • Problem

    Reward misspecification in the high-reward region can cause reward over-optimization, while obtaining sufficient high-reward examples under the base LLM is difficult.

  • Method

    The paper constructs rubric-based rewards from off-policy exemplars and refines rubrics to distinguish among high-quality responses.

  • Results

    Rubric-based rewards are empirically highly effective for LLM post-training, and correctly ranking a small top proportion of responses can approach optimal performance at moderate KL.

  • Takeaways & Limitations

    Accurate discrimination in the high-reward tail is the critical target for mitigating reward over-optimization in post-training.

  • Takeaways & Limitations

    The study uses simple weighted-average rubric aggregation and notes that larger-scale Bradley-Terry training data may be needed for comparable generalizability, especially in specialized domains.

Abstract

from arXiv · show

Reinforcement fine-tuning (RFT) often suffers from reward over-optimization, where a policy model hacks the reward signals to achieve high scores while producing low-quality outputs. Our theoretical analysis shows that the key lies in reward misspecification at the high-reward tail: the inability to reliably distinguish Excellent responses from merely Great ones. This motivate us to focus on the high-reward region. However, such tail examples are scarce under the base LLM. While off-policy exemplars (e.g. from stronger models or rewrites) are easier to obtain, naively training on them yields a misspecified reward for the policy we aim to align. To address this, we study rubric-based rewards. By design, rubrics can leverage off-policy examples while remaining insensitive to their artifacts. To elicit rubrics that capture the high-reward tail, we highlight the importance of distinguishing among great and diverse responses, and introduce a workflow to implement this idea. We empirically demonstrate that rubric-based rewards substantially mitigate reward over-optimization and deliver effective LLM post-training improvements.

1 INTRODUCTION

Reward misspecification matters most in the high-reward region, yet high-quality training examples are rare under the base LLM. The paper therefore develops rubric-based rewards that use off-policy exemplars while targeting distinctions among the best responses.

  • 1 INTRODUCTION: High-reward-region errors dominate reward misspecification effects for Pareto-optimal post-training procedures.The analysis identifies accurate discrimination among very good responses as the key requirement.
  • 1 INTRODUCTION: Base-LLM sampling is inefficient for obtaining high-reward examples, whereas off-policy generation supplies such examples more readily but can misalign the learned reward.Examples may come from stronger models, extensive reasoning, or rewrites.
  • 1 INTRODUCTION: Rubric-based rewards use off-policy exemplars to construct criteria that remain insensitive to irrelevant response artifacts.The paper studies how to elicit rubrics that capture the high-reward tail and evaluates a workflow for doing so.
  • 1 INTRODUCTION: The paper contributes a theoretical characterization of high-reward misspecification, an off-policy rubric-construction method, and empirical evidence for rubric efficacy.These contributions also examine the critical role of high-reward-region misspecification.

2 PRELIMINARIES

The paper formulates reinforcement fine-tuning around proxy rewards and KL-regularized policy optimization, then introduces rubric-based rewards as an interpretable alternative for tasks without easily verifiable outcomes. Existing proxy-reward methods remain vulnerable to reward over-optimization.

  • 2 PRELIMINARIES: A reward model scores prompt-response pairs, while π, π0, r⋆, and r denote the policy, reference model, gold reward, and practical proxy reward.The gold reward is inaccessible in practice.
  • 2 PRELIMINARIES: RFT optimizes expected reward while penalizing KL divergence from the reference model, with β controlling deviation from that reference.The objective is defined over prompts sampled from D and responses sampled from the policy.
  • 2 PRELIMINARIES: Reward over-optimization occurs when a policy exploits proxy-reward inaccuracies, raising proxy scores while true quality deteriorates.Online RLHF can mitigate this through periodic fresh human feedback, but it is costly and slow.
  • 2 PRELIMINARIES: Rubric-based rewards associate each prompt with weighted explicit criteria, and a verifier scores whether each response satisfies those criteria.The total reward is computed from the weighted criteria that the response meets.
  • 2 PRELIMINARIES: RLRR extends verifiable-reward reinforcement learning to general tasks, but whether it alleviates reward over-optimization remains unclear.Its explicit criteria make rewards more interpretable and harder to game than Bradley-Terry reward models.

3 HIGH-REWARD REGION ACCURACY IS KEY TO OVERCOMING REWARD OVER-OPTIMIZATION

The analysis models reward misspecification as a mapping from gold to proxy rewards and shows that errors in the high-reward region dominate post-training performance. Experiments and the proposed iterative rubric-refinement workflow therefore prioritize accurately ranking and differentiating top-quality responses.

  • Theoretical analysis: The analysis uses a mapping f from ground-truth rewards r⋆ to proxy rewards r to study how misspecification geometry affects utility-KL tradeoffs.The framework examines the relationship between utility, including expected reward and win rate, and KL divergence under misspecified rewards.
  • Theoretical analysis: High-reward misspecification dominates the utility-KL tradeoff because exponential penalties intensify as reward values increase.The KL divergence remains invariant to the misspecification mapping, while the exponential term imposes increasingly severe penalties in the high-reward regime.
  • Theoretical analysis: Accurately ranking a small top proportion of responses can approach the optimal win-rate curve despite errors elsewhere.The supplied analysis reports near-optimal behavior when a sufficiently large top proportion is correctly identified and ordered, including examples of 10% and 40%.
  • Theoretical analysis: High-reward inaccuracies can produce acceptable performance at small KL but collapsing win rates as KL grows.This pattern is identified across misspecification settings and is linked to reward over-optimization behavior.
  • Rubric refinement: Iterative rubric refinement scores candidate responses, compares the top two, and encodes their distinguishing features into a final rubric set.Repeated rounds progressively focus rubric discovery on higher-quality candidates through filtering and response differentiation.

4 PRINCIPLES FOR CONSTRUCTING RUBRICS

The section argues that effective rubrics must resolve distinctions within the high-reward tail, especially between excellent and merely great responses, and operationalizes this through iterative refinement over off-policy candidates.

  • 4 PRINCIPLES FOR CONSTRUCTING RUBRICS: Rubric restrictions reduce sensitivity to irrelevant off-policy artifacts but can make excellent and merely great solutions tie.
  • 4 PRINCIPLES FOR CONSTRUCTING RUBRICS: Iterative refinement with great pairs outperforms refinement with good pairs, while diverse great pairs produce further downstream RL gains.
  • 4 PRINCIPLES FOR CONSTRUCTING RUBRICS: Effective rubric construction requires distinguishing excellent responses from great ones and distinguishing among diverse off-policy responses.
  • 4.1 METHODOLOGY: Refinement-through-Differentiation prompts a proposer LLM to compare candidate responses and encode their distinguishing features as rubric criteria or refinements.
  • 4.1 METHODOLOGY: Repeated refinement scores an off-policy candidate pool, selects the top two responses, and applies RTD to concentrate discovery on the performance frontier.
  • 4.2 EXPERIMENTAL SETUP: Experiments evaluate reward over-optimization and capability gains across general-purpose, healthcare, and finance domains.
  • 4.2 EXPERIMENTAL SETUP: The synthetic oracle setting uses a strong LLM both to generate reward-modeling rubric data and to judge held-out performance, reducing annotation-evaluation confounds.
  • 4.2 EXPERIMENTAL SETUP: Candidate-pool comparisons test better versus weaker model pairs for Principle 1 and 16 responses from diverse excellent models for Principle 2.

5 RESULTS

Refining rubrics with better and more diverse responses improves downstream RL, sustains performance during prolonged training, and sharpens reward accuracy in the high-reward tail. Great-response refinements also produce more sophisticated rubric criteria than good-response refinements.

  • 5.1 RL IMPROVES WITH BETTER AND MORE DIVERSE RESPONSES: Great-pair refinement outperforms good-pair refinement, while iterative refinement with multiple diverse great pairs yields further downstream RL gains.These results validate both rubric-construction principles.
  • 5.1 RL IMPROVES WITH BETTER AND MORE DIVERSE RESPONSES: Iteratively refined rubrics sustain higher win rates and benchmark scores longer, delaying reward over-optimization during extended training.Initial rubrics and single-pair refinements peak early before declining; over-optimization appears only at later stages with diverse iterative refinement.
  • 5.2 REWARD MODEL ACCURACY IMPROVES IN THE HIGH-REWARD TAIL: Great-pair refinements mainly improve high-reward accuracy, whereas good-pair refinements improve low-reward accuracy more; diverse great-pair iteration pushes high-reward accuracy further.Rubric accuracy was evaluated against ground-truth preferences on 1,000 training prompts.
  • 5.3 REFINEMENTS FROM BETTER RESPONSES ARE MORE SOPHISTICATED: Great responses more often induce sophisticated refinements, including decomposed criteria and stronger verification standards, while good responses drive basic corrections.The comparison uses great Gemini 2.5 Pro pairs and good Gemini 2.5 Flash Lite pairs in healthcare.
  • 5.3 REFINEMENTS FROM BETTER RESPONSES ARE MORE SOPHISTICATED: A medical example resolves a tie between great responses by adding a criterion requiring urgent imaging to confirm the diagnosis.The added criterion comes from the cluster on enhancing verification, validation, and evidence standards.

6 RELATED WORK

Prior work documents reward over-optimization and rubric-based rewards, while this paper focuses on the high-value region as the decisive locus of proxy-reward fidelity. It positions rubric rewards as an established approach across specialized, general, and agentic tasks.

  • Reward over-optimization: The paper sharpens reward-over-optimization analysis by arguing that proxy fidelity in the high-value region governs post-training performance more than global deviation statistics.Existing analyses typically relate degradation to global proxy-reward mismatch.
  • Rubrics reward: Rubric-based rewards have been applied to specialized science and health domains, general instruction following, and agentic ability using online and offline RL.

7 DISCUSSION

The discussion frames rubric rewards as a response to the sample inefficiency of off-policy Bradley-Terry training and acknowledges that score aggregation remains unresolved. The method's scope is therefore strongest where limited data can encode generalizable principles, but its aggregation is deliberately simple.

  • 7 DISCUSSION: Bradley-Terry reward models may require very large, diverse off-policy datasets, whereas rubric rewards can encode generalizable principles from limited specialized-domain data.The paper notes that 5,000 off-policy responses did not help guide the current policy, while prior work used up to 20 million samples.
  • 7 DISCUSSION: The weighted-average aggregation used for rubric scores is not considered optimal, and designing better aggregation is left for future work.

USAGE OF LARGE LANGUAGE MODELS

The paper uses LLMs for purposes beyond running them in experiments.

  • USAGE OF LARGE LANGUAGE MODELS: The paper uses LLMs for purposes beyond running them in experiments.

1. Aid or Polish Writing (Gemini 2.5 Pro, ChatGPT 4/5)

The passage identifies literature retrieval and discovery, such as finding related work, as a research-support use of LLMs.

  • LLMs can assist with literature retrieval and discovery, including finding related work.

3. Assisting Code Writing and Debugging (Claude-Ops-4.1, GPT-5)

The passage emphasizes monitoring LLM use in academic research for plagiarism and fabricated facts, while asserting that these problems were prevented in the paper.

  • Academic LLM use was monitored for plagiarism and fabricated facts, with the authors stating these problems did not occur.

A THEORETICAL RESULTS

The theoretical results identify high-reward misspecification as especially consequential while describing rubric construction principles and empirical evidence that Bradley-Terry models are difficult to improve with moderate off-policy data.

  • Theoretical motivation: Misspecification in the high-reward region has disproportionately large effects on the utility-KL tradeoff, while KL divergence remains invariant to reward transformation.The exponential term imposes increasingly severe penalties for high-reward misspecification relative to low-reward misspecification.
  • Rubric refinement: Rubric refinement targets insufficient discrimination among high-quality responses by adding, rewriting, decomposing, or deleting criteria while preserving comprehensive and consistent evaluation.The refinement prompt explicitly addresses rubrics that cannot effectively differentiate high-quality responses.
  • Bradley-Terry reward modeling: Adding off-policy and on-policy responses improves win rates but not HealthBench, suggesting superficial reward-model features rather than true capabilities.The passage presents on-policy responses as a baseline that is difficult to improve and characterizes the experiment as showing the difficulty of using off-policy responses.

E LLM JUDGE FOR EVALUATION

The evaluation setup uses GPT-4.1 to propose and judge rubric-related responses, controls position bias, and analyzes rubric capabilities across curated prompts and refined examples.

  • Evaluation design: Using the same GPT-4.1 model to propose rubrics and judge outputs isolates candidate-response quality as the experimental variable.This design avoids confounding from disagreements between rubric proposer and evaluator.
  • Prompt selection: Candidate prompts are curated for clear intent, generalist accessibility, nontriviality, and an objectively better response, excluding multimodal, search, trivia, and creative-writing tasks.
  • Rubric refinement analysis: Rubric refinement differences are decomposed into atomic improvements and clustered to characterize recurring refinement patterns.The clustering uses differences from refinements based on one-good-pair and one-great-pair examples.
  • Rubric refinement analysis: A medical example shows refinement resolving a tie between two great responses by adding an imaging-test criterion, thereby strengthening verification and evidence standards.
  • Capability analysis: The constructed rubrics target diverse, concrete capabilities, with hierarchical clustering identifying 20 capability types across health-domain rubric criteria.Examples include logical structuring, evidence-based justification, risk assessment, strategic planning, and sequential quantitative procedures.
Loading 2509.21500v3…