PaperlayerView exact revision on arXiv

Source-linked AI summary

CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling

Dengcan Liu, Fengkai Yang, Xiaohan Wang, Shurui Yan, Jiajun Chai, Jiahao Li, Yikun Ban, Zhendong Mao, Wei Lin, Guojun Yin

arXiv:2603.08035v1cs.AIcs.LG

TL;DR

CDRRM addresses opaque, annotation-intensive reward modeling and unreliable rubric generation by contrasting preference pairs to identify discriminative factors, then synthesizing compact rubrics for guided judgments. Across RewardBench, RMBench, and RMB, it achieves state-of-the-art performance and shows data efficiency, including frozen-base-model performance exceeding fully fine-tuned baselines.

  • Problem

    Existing reward models are opaque and annotation-intensive, while rubric-based methods can produce noisy, redundant criteria and fail to mitigate evaluator biases.

  • Method

    CDRRM performs multi-dimensional contrastive profiling on preference pairs and synthesizes the resulting evidence into concise, context-aware rubrics that guide preference judgments.

  • Results

    CDRRM achieves state-of-the-art performance across RewardBench, RMBench, and RMB; average accuracy improves by 5.7% over rubric-based baselines, with an 18% gain on RMBench Hard.

  • Takeaways & Limitations

    Training on only 3k high-quality samples enables a frozen base model guided by synthesized rubrics to outperform fully fine-tuned baselines.

Abstract

from arXiv · show

Reward modeling is essential for aligning Large Language Models(LLMs) with human preferences, yet conventional reward models suffer from poor interpretability and heavy reliance on costly expert annotations. While recent rubric-based approaches enhance evaluation transparency, they lack systematic quality control, yielding noisy and redundant criteria, failing to mitigate persistent biases (e.g., verbosity, position) in LLM evaluators, and creating a scalability-reliability trade-off. To address these limitations, we propose CDRRM (Contrast-Driven Rubric Reward Model), a framework built on a novel Contrast-then-Synthesis paradigm for high-quality rubric generation and guided preference judgment. CDRRM first conducts multi-dimensional contrastive profiling on preference pairs to identify causal discriminative factors, then synthesizes these insights into compact, context-aware rubrics to guide preference judg- ments. Extensive experiments on three authoritative benchmarks (RewardBench, RMBench, RMB) demonstrate that CDRRM achieves state-of-the-art performance across diverse domains and effectively mitigates aforementioned evaluation biases. Notably, our approach delivers exceptional data efficiency: training the rubric generator on only 3k high-quality samples empowers a frozen pre-trained judge model to outperform fully fine-tuned baselines. This work offers a scalable, interpretable, and data-efficient path for reward modeling.

1 Introduction

CDRRM addresses opaque, annotation-intensive reward modeling and unreliable rubric generation with contrastive profiling followed by rubric synthesis. Across three benchmarks, it improves performance and mitigates evaluator biases while requiring only limited training data.

  • Traditional scalar reward models are opaque and depend heavily on costly, large-scale expert annotations.
  • Existing rubric-generation methods rely on manual annotation or direct prompting, producing noisy, redundant criteria that may not reflect discriminative factors.
  • The framework performs contrastive analysis of preference pairs to identify causal discriminative factors before synthesizing concise, context-aware rubrics.
  • CDRRM is evaluated on RewardBench, RMBench, and RMB, achieving state-of-the-art performance across diverse domains while mitigating verbosity and position biases.
  • Training the Rubric Generator on only 3k high-quality samples enables a frozen base model guided by synthesized rubrics to outperform fully fine-tuned baselines.
  • CDRRM improves average accuracy by 5.7% across benchmarks and achieves an 18% gain on RMBench Hard over rubric-based baselines.

2 Preliminaries

The preliminaries motivate rubric learning as a transparent alternative to opaque reward functions, while showing that directly prompted rubrics are often redundant and noisy. Pruning experiments indicate that many existing criteria contribute little to accuracy.

  • Rubric learning organizes response evaluation into criteria spanning multiple quality dimensions, making reward computation more transparent and interpretable.
  • Directly prompted rubrics lack fine-grained alignment with discriminative human standards, producing overlapping semantics and irrelevant details that can misguide training.
  • Most existing rubric datasets contain at least seven rubrics per sample, despite preference judgments typically depending on a sparse set of salient factors.
  • Pruning one to three rubrics causes at most 0.42% validation accuracy deviation, indicating substantial redundancy in generated criteria.
  • The observed redundancy motivates a more selective rubric-generation process based on Contrast-then-Synthesis.

3 Methodology

CDRRM generates rubrics by contrasting chosen and rejected responses across task-relevant dimensions, anchoring judgments in evidence, and synthesizing concise criteria. Consistency filtering then produces training data for a Rubric Generator and rubric-guided Judge Model.

  • Contrast-then-Synthesis combines multi-dimensional Contrastive Profiling with Rubric Synthesis to isolate preference-driving factors and produce concise evaluation criteria.
  • 3.1 Contrastive Profiling: The method dynamically selects evaluation dimensions relevant to each instruction-response pair instead of applying a static taxonomy.
  • 3.1 Contrastive Profiling: Evidence-Anchored Constraint requires judgments for active dimensions to be grounded in original text spans, improving the interpretability and discriminability of synthesized rubrics.
  • 3.2 Rubric Synthesis: Rubric Synthesis conditions rubric generation on explicit contrastive profiles, aligning criteria with observed quality gaps and reducing redundant rubrics from model priors.
  • 3.3 Consistency Filtering and Dataset Construction: Preference-Consistency filtering retains only rubric sets whose rubric-guided preference prediction matches the ground-truth label, forming the Drubric training dataset.
  • 3.4 Model Training: The framework trains a Rubric Generator on validated rubric data and a Judge Model on rubric-conditioned preference justifications.

4 Experiments

CDRRM is evaluated on three reward-modeling benchmarks using contrastive rubric generation, with experiments spanning data construction, model variants, baselines, and bias robustness.

  • Datasets and Experiment Settings: Training constructs separate 3,000-instance datasets for Rubric Generator targets and rubric-grounded Judge Model preference judgments.The Rubric Generator uses contrastive profiling, while the Judge Model dataset contains rubric-conditioned justifications and preference labels.
  • Datasets and Experiment Settings: Experiments use RewardBench, RMBench, and RMB, with accuracy defined as the proportion of preference pairs where the chosen response is correctly identified.The study evaluates CDRRM against scalar, generative, and rubric-based reward-modeling paradigms.
  • Main Results: CDRRM-14B (SFT) achieves an average score of 88.3, improving 5.7% over RM-R1-Qwen-Instruct-32B and 3.6% over BR-RM-Qwen-8B.CDRRM uses only 3k samples each to train the Rubric Generator and Judge Model.
  • Main Results: CDRRM-8B (Base) reaches 85.8 without Judge Model fine-tuning, exceeding fully fine-tuned BR-RM-Qwen-8B at 85.2 and RM-R1-Qwen-Instruct-32B at 83.5.The base judge is prompted only with rubrics produced by the Rubric Generator.
  • Robustness against Biases: On RMBench Hard, CDRRM-8B (Base) scores 81.1 and CDRRM-14B (SFT) scores 83.4, above the best scalar, generative, and rubric-based baselines.RMBench Hard evaluates sensitivity to subtle content differences and resistance to verbosity and position biases.
  • Robustness against Biases: CDRRM conditions judgments on structured criteria to shift attention from superficial verbosity or position cues toward fine-grained quality distinctions.The reported robustness gains are strongest on the complex cases represented by RMBench Hard.

5 Analysis

The analysis tests CDRRM’s rubric mechanism through ablations, data-scaling experiments, and a verbosity-bias case study. Results show benefits from contrastive rubric generation, rapid performance saturation, and improved robustness against superficial length cues.

  • 5.1 Ablation Study: CDRRM outperforms the One-step Rubric Judge, while Direct Judge significantly underperforms both rubric-based approaches.The comparison attributes the gap to rubrics grounded in fine-grained contrastive profiles rather than generic model priors.
  • 5.2 Scaling Analysis: 85.6 average score with 1k Rubric Generator samples rises only to 86.0 with 12k, indicating rapid performance saturation.The scaling experiment evaluates an untuned Qwen3-8B model and reports the trend in Figure 3(a).
  • 5.2 Scaling Analysis: Judge Model performance improves from 1k to 3k training samples before entering a plateau beyond 3k, even up to 20k samples.The authors attribute this saturation to explicit discriminative rubrics simplifying the preference-prediction objective.
  • 5.3 Case Study: In the verbosity-bias case, Direct Judge prefers a lengthy truncated response, whereas CDRRM penalizes truncation using completeness and conciseness rules.The case study illustrates how explicit rubrics counter superficial preferences for length and formatting.

6 Related Works

Related work progresses from opaque scalar reward models toward interpretable generative and rubric-based evaluation. However, rubric construction and reliable LLM-as-a-Judge behavior remain unresolved challenges.

  • Reward Modeling: Traditional scalar reward models provide ranking but remain opaque and lack explicit reasoning.Generative reward models address this limitation through techniques such as chain-of-thought or critique generation.
  • Rubric-Based Methods: Rubric-based methods structure evaluation reasoning, but a persistent gap remains between rubric generation and effective optimization.This gap motivates improving how rubrics are constructed and aligned with preference judgments.
  • LLM-as-a-Judge: LLM-as-a-Judge offers scalable evaluation across domains, yet systematic biases continue to limit its reliability.The related-work discussion identifies reliability as a critical bottleneck despite broad adoption.

7 Conclusion

The paper concludes that CDRRM combines contrastive rubric synthesis with explicit preference guidance to improve reward-model interpretability, performance, data efficiency, and bias robustness.

  • Conclusion: CDRRM uses Contrast-then-Synthesis to construct task-aligned rubrics that anchor preference decisions to explicit criteria.The framework is presented as mitigating opacity in traditional reward modeling.
  • Conclusion: Frozen base models guided by CDRRM can outperform fully fine-tuned state-of-the-art baselines with limited training samples.The conclusion also reports qualitative evidence that the framework alleviates prevalent biases, particularly verbosity bias.

A Experiment Setups

The experiments evaluate reward models on RewardBench, RM-Bench, and RMB, which cover complementary capabilities and scenarios. Accuracy is used as the core evaluation metric.

  • Benchmarks: RewardBench evaluates prompt-chosen-rejected trios across chat, chat-hard, reasoning, and safety categories.The benchmark is described as foundational for reward-model evaluation.
  • Benchmarks: RM-Bench tests sensitivity to subtle content differences and robustness against style-related biases across chat, safety, math, and code.Its prompts vary in difficulty and impose higher demands on fine-grained judgment.
  • Benchmarks: RMB assesses helpfulness and harmlessness across more than 49 real-world scenarios, with pairwise and Best-of-N formats and 25,845 instances.Its broader scenario coverage complements the narrower category structures of RewardBench and RM-Bench.
  • Evaluation Metric: All experiments use accuracy, defined as the proportion of preference pairs whose chosen response is correctly identified over the rejected response.The benchmark specifications and further details are provided in the appendix.

B Full Experiment Results

This section supplements the main findings with complete experimental results and broader baseline comparisons. RewardBench results are provided in Table 6.

  • The appendix reports complete experimental results and a comprehensive comparison with existing baselines.
  • Full results on RewardBench are presented in Table 6.
  • The section is positioned as a supplement to the key findings reported in the main text.

C Prompts

This section documents the prompt templates used throughout CDRRM. It covers contrastive profiling, rubric synthesis, rubric-generator training, and judge-model training.

  • The section provides complete prompt templates for CDRRM’s core steps and training pipelines.
  • System prompts are supplied for Contrastive Profiling and Rubric Synthesis, the two components of the Contrast-then-Synthesis paradigm.
  • Additional templates are provided for training the Rubric Generator and Judge Model, CDRRM’s two core modules.

D Case Study

The appendix details CDRRM’s rubric-generation prompts and examines difficult reward-modeling cases. Its case studies compare CDRRM with direct and one-step rubric-guided judgments on verbosity, naming, and geometric reasoning errors.

  • Case Study: CDRRM’s case studies report correct judgments on verbosity bias, subtle naming errors, and geometric modeling errors.The comparisons cover representative RM-Bench samples in code and mathematical/reasoning scenarios.
  • Case Study: In the verbosity case, CDRRM generates hard rules for completeness and structural integrity, selecting concise complete Response A over flawed Response B.
  • Case Study: In the naming-error case, CDRRM targets instruction-aligned naming rules and identifies Response A’s compliance over Response B’s incorrect function name.
  • Case Study: In the geometric-reasoning case, CDRRM targets valid geometric modeling and answer-configuration consistency to identify Response A as correct.
  • Rubric Generation: The rubric generator is instructed to produce hard rules and principles, with each rule containing an identifier, type, criterion, test, and rationale.
  • Contrastive Profiling: Contrastive profiling diagnoses answer quality across instruction-linked dimensions using verifiable evidence and explicit instruction anchoring.
Loading 2603.08035v1…