PaperlayerView exact revision on arXiv

Source-linked AI summary

C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences

Akira Kawabata, Saku Sugawara

arXiv:2604.13618v1cs.CLcs.LG

TL;DR

Rubric-augmented verification is difficult to scale because existing methods depend on costly rubric annotations, and poor self-generated rubrics can mislead reward models. C2 learns a cooperative generator and critical verifier from binary preferences, improving preference prediction and RLHF outcomes over reasoning reward-model baselines.

  • Problem

    Existing rubric-augmented verification methods rely on costly external rubric annotations, while low-quality self-generated rubrics can mislead reward models.

  • Method

    C2 synthesizes helpful and misleading rubric pairs from binary preferences to train a cooperative generator and critical verifier that selectively follows helpful rubrics.

  • Results

    C2 outperforms GRPO-trained reasoning reward models, gaining +6.5 points on RM-Bench and +6.0 points LC win rate on AlpacaEval.

  • Takeaways & Limitations

    C2 realizes rubric-augmented verification from binary preferences alone and supports stronger aligned policies through RLHF.

  • Takeaways & Limitations

    C2 depends on base-model reasoning capability and adds computational overhead from rubric generation and possible rubric-free retry inference.

Abstract

from arXiv · show

Rubric-augmented verification guides reward models with explicit evaluation criteria, yielding more reliable judgments than single-model verification. However, most existing methods require costly rubric annotations, limiting scalability. Moreover, we find that rubric generation is vulnerable to a failure of cooperation; low-quality rubrics actively mislead reward models rather than help. Inspired by the principle of cooperative communication, we propose Cooperative yet Critical reward modeling (C2), a framework that significantly improves reward model judgments by having the reward model critically collaborate with a rubric generator trained solely from binary preferences. In C2, we synthesize helpful and misleading rubric pairs by measuring how each rubric shifts the reward model toward or away from the correct preference. Using these contrastive pairs, we train a cooperative rubric generator to propose helpful rubrics, and a critical verifier to assess rubric validity before making its judgment, following only rubrics it deems helpful at inference time. C2 outperforms reasoning reward models trained on the same binary preferences, with gains of up to 6.5 points on RM-Bench and 6.0 points length-controlled win rate on AlpacaEval 2.0. Without external rubric annotations, C2 enables an 8B reward model to match performance achieved with rubrics from a 4$\times$ larger model. Overall, our work demonstrates that eliciting deliberate cooperation in rubric-augmented verification makes reward models more trustworthy in a scalable way.

1 Introduction

C2 addresses the scalability and reliability challenges of rubric-augmented verification by learning rubric generation and rubric assessment from binary preferences. It uses cooperative generation and critical verification to improve reward-model judgments.

  • Existing rubric-augmented methods often require costly human or proprietary-model annotations, limiting scalability and reuse of binary preference data.
  • Self-generated rubrics vary in quality: helpful rubrics improve judgments, while vague or misaligned rubrics can actively mislead verifiers.
  • C2 frames rubric generation and verification as cooperative communication in which generators learn which rubrics help and verifiers learn which rubrics to trust.
  • C2 synthesizes contrastive helpful–misleading rubric pairs from binary preferences, training a cooperative generator and a critical verifier with selective inference.
  • 6.5 points on RM-Bench and 6.0 points LC win rate on AlpacaEval show gains over GRPO-trained reasoning reward models.

2 Related Work

Related work uses rubrics to make verification more interpretable and reliable, but scalability and rubric correctness remain unresolved challenges. C2 addresses both by deriving rubrics from binary preferences and assessing them before use.

  • Rubric-augmented verification decomposes holistic evaluation into fine-grained criteria for improved interpretability and reliability.
  • Existing approaches commonly obtain rubrics from human annotators or proprietary models, limiting scalability.
  • Many current methods assume rubric correctness and overlook incomplete or misleading rubrics.
  • C2 derives rubrics from binary preferences alone and trains the verifier to assess rubric quality before following each rubric.

3 Do Self-Generated Rubrics Help Verification?

The experiments test whether self-generated rubrics help pairwise preference verification and show that rubric quality, rather than rubric presence alone, determines their value. Helpful rubrics improve accuracy, whereas misleading ones substantially reduce it.

  • 3.1 Experimental Setup: The study evaluates pairwise preference prediction, where a verifier selects the better response from two candidates given a prompt.
  • 3.1 Experimental Setup: RM-Bench’s hard subset pairs stylistically favorable rejected responses with less polished chosen responses, testing whether rubrics focus verification on substance over style.
  • 3.1 Experimental Setup: Each rubric contains reasoning explaining its checklist plus criterion–question pairs with yes/no questions.
  • 3.2 Experiment 1: Overall Effect of Self-Generated Rubrics: The experiment measures each rubric’s shift in verifier probability assigned to the gold label, where positive values move toward and negative values away from the correct decision.
  • 3.2 Experiment 1: Overall Effect of Self-Generated Rubrics: Most self-generated rubrics barely affect confidence, and naive self-generation offers little benefit over rubric-free verification.
  • 3.3 Experiment 2: Impact of Rubric Quality: Random rubrics remain near the no-rubric baseline: Tulu3 changes from 50.3% to 48.3%, while Qwen3 changes from 61.0% to 62.4%.
  • 3.3 Experiment 2: Impact of Rubric Quality: High-quality rubrics raise accuracy to 58.5% (+8.2) for Tulu3 and 74.7% (+13.6) for Qwen3, while low-quality rubrics reduce it to 39.6% and 49.3%.

4 C2: Cooperative yet Critical Reward Modeling

C2 trains a cooperative rubric generator and critical verifier using contrastive helpful-versus-misleading rubrics derived from binary preferences. At inference, the verifier follows only rubrics it judges helpful and otherwise evaluates without a rubric.

  • 4.1 Synthesizing Helpful and Misleading Rubrics: C2 labels self-generated rubrics by measuring whether they shift verifier judgments toward or away from the gold preference.The base model serves as both rubric generator and verifier, with rubric-free judgments providing the comparison baseline.
  • 4.2 Training the Generator and Verifier: The generator is trained with DPO, while the verifier predicts preferences and rubric quality using GRPO-based rewards.The rubric-augmented task adds a rubric-assessment reward to format and preference rewards.
  • 4.1 Synthesizing Helpful and Misleading Rubrics: The method samples K = 16 rubric candidates and computes each candidate’s verifier margin relative to the rubric-free task.Rubrics are retained according to whether they improve correct predictions or worsen incorrect ones.
  • 4.1 Synthesizing Helpful and Misleading Rubrics: Helpful rubrics must produce correct predictions, while misleading rubrics must push the verifier toward incorrect predictions.The thresholds require correctness rather than merely improvement over the rubric-free baseline.
  • 4.1 Synthesizing Helpful and Misleading Rubrics: C2 selects the strongest helpful and misleading rubrics as contrastive outputs, discarding examples lacking either set.These contrastive pairs supervise both rubric-generator and verifier training.
  • 4.3 Selective Rubric Use: At inference, the verifier returns its rubric-conditioned judgment only when it labels the rubric helpful; otherwise, it falls back to rubric-free evaluation.This selective policy lets the verifier reject rubrics it does not trust.

5 Experiments

Experiments compare C2 with reasoning reward models and rubric-based baselines on preference prediction and downstream policy optimization. C2 is evaluated across multiple benchmarks, models, and compute settings.

  • 5 Experiments: The experiments measure preference prediction against reasoning reward models and naive self-rubric augmentation, then test downstream policies trained with DPO.The evaluation covers both reward-model accuracy and whether improved preference prediction transfers to policy optimization.
  • 5.1 Reward Modeling: Baselines include the base model, Reasoning RM, Reasoning RM + Self-Rubric, and external rubrics generated by Qwen3-32B.The external-rubric setting uses a substantially larger model and serves as an upper-bound comparison.
  • 5 Experiments: C2 uses 5,000 rubric-free instances plus rubric-augmented instances built from retained helpful and misleading rubrics.Retention requires at least one helpful and one misleading rubric per example.
  • 5 Experiments: The evaluation covers RewardBench, RM-Bench, RewardBench2, and JudgeBench for preference prediction, plus AlpacaEval 2.0 and Arena-Hard for downstream alignment.The benchmarks target settings where evaluation criteria are implicit and correctness is not readily verifiable.
  • 5.1 Reward Modeling: 58.3% average accuracy gives C2 a +3.3-point gain over Reasoning RM for Tulu3-8B-SFT, while Qwen3-8B reaches 78.5% and matches the Qwen3-32B external-rubric setting.On RM-Bench, C2 exceeds Reasoning RM by 6.5 points, 87.8% versus 81.3%.
  • 5.2 RLHF Performance: C2 consistently outperforms Reasoning RM on downstream benchmarks, with larger gains for Tulu3 than Qwen3.The reported gains are 6 and 5.5 points for Tulu3, versus 2.7 and 2.8 points for Qwen3, across the stated benchmarks.

6 Analysis

The analysis tests whether C2’s gains arise from its cooperative–critical design, its robustness to noisy rubrics, improved rubric generation, and its component choices. Across these tests, C2 outperforms compute-matched reasoning reward models, remains more stable under noisy rubrics, improves rubric quality, and benefits most from misleading-rubric supervision.

  • 6.1 Does C2 Simply Benefit from More Compute?: C2 consistently outperforms compute-matched Reasoning RM across all generation budgets.It maintains a 2–3 point advantage for Tulu3-8B and approximately 2 points for Qwen3 across all N.
  • 6.2 How Robust Is C2 to Noisy Rubrics?: Reasoning RM accuracy drops from 53% to 39% for Tulu3-8B-SFT and from 73% to 52% for Qwen3-8B as low-quality rubrics increase.The comparison spans the 9:1 to 1:9 high-quality-to-low-quality rubric conditions.
  • 6.2 How Robust Is C2 to Noisy Rubrics?: C2 accuracy decreases only from 51% to 46% for Tulu3-8B-SFT and from 76% to 70% for Qwen3-8B over the same rubric range.Selective inference provides increasing benefits as low-quality rubrics become more prevalent, although Reasoning RM slightly outperforms C2 for Tulu3-8B-SFT at 9:1.
  • 6.3 How Much Does Generator Training Improve Rubric Quality?: DPO training shifts rubric-quality distributions toward higher scores, raising mean scores from 2.11 to 2.66 for Tulu3-8B and from 3.15 to 3.52 for Qwen3-8B.The resulting means narrow the gaps to larger same-family models, whose scores are 2.85 and 3.62, respectively.
  • 6.4 Ablation Study: All C2 components contribute, with negative rubrics most critical and the critical verifier contributing more than the cooperative generator.Removing misleading rubrics causes the largest drop, while the comparison indicates selective rubric trust matters more than generation alone.

7 Conclusion

C2 addresses the two-sided quality of self-generated rubrics by learning cooperation between a rubric generator and a critical verifier. The framework improves verification and aligned-policy performance without external rubric annotations, while its gains persist beyond increased inference compute.

  • 7 Conclusion: C2 realizes rubric-augmented verification from binary preferences alone by synthesizing contrastive rubrics and training a cooperative generator with a critical verifier.The framework uses confidence shifts to identify helpful and misleading rubrics.
  • 7 Conclusion: High-quality rubrics improve verification, whereas low-quality rubrics degrade performance below the rubric-free baseline.C2 uses this two-sided behavior to train its generator and verifier on contrastive signals.
  • 7 Conclusion: C2 outperforms GRPO-trained reasoning reward models on four preference benchmarks, with gains translating to stronger aligned policies through RLHF.The analysis attributes these gains to C2’s design rather than increased inference compute, and selective inference remains robust when most rubrics are misleading.

Limitations

The paper identifies dependence on base-model reasoning capability and additional computational overhead as its two main limitations. Weaker models may reject useful rubrics, while generation and retry steps increase inference cost.

  • Limitations: C2’s effectiveness depends on the base model’s reasoning capability.Weaker models may struggle to distinguish helpful from misleading rubrics and unnecessarily reject useful guidance.
  • Limitations: C2 incurs additional computational overhead because rubric generation precedes verification and retries may trigger rubric-free inference.The paper suggests more efficient rubric generation or selective rubric use as ways to broaden applicability in resource-constrained settings.

B Implementation Details

The implementation trains verifiers with GRPO and rubric generators with DPO, evaluates downstream policies with DPO, and uses weighted format, preference, and rubric rewards. C2 requires substantially more inference tokens because of rubric generation and possible retries.

  • Training setup: The verifier uses GRPO training, while the rubric generator uses DPO training with shared generator hyperparameters across Tulu3-8B and Qwen3-8B.Reasoning RM and C2 share GRPO settings except for training epochs; Reasoning RM uses 3 epochs and C2 uses 1.
  • Downstream RLHF: Downstream RLHF experiments use the same DPO hyperparameters to fine-tune policies on preference pairs constructed by C2 and Reasoning RM.For Qwen3-8B, thinking is disabled during response sampling and policy evaluation because enabling it caused significant performance degradation.
  • Compute and software: Experiments use trl for DPO and GRPO training, vLLM for inference, and 8 NVIDIA A100 80GB GPUs.
  • Inference cost: C2 generates 1,862 versus 803 average tokens per example for Tulu3-8B-SFT and 2,465 versus 1,018 for Qwen3-8B.These correspond to 2.3× and 2.4× the Reasoning RM token consumption, respectively.
  • Rubric-pair sampling: Rubric-pair sampling draws K = 16 candidates and retries up to 5 additional times when helpful and misleading rubrics are not both obtained.Examples are discarded only if no pair can be formed after the retries.

D Additional RLHF Experiments

C2’s reward-model gains transfer to best-of-N selection and rejection sampling. It consistently outperforms Reasoning RM across the evaluated benchmarks and sampling settings.

  • Best-of-N selection: C2 consistently outperforms Reasoning RM across all best-of-N benchmarks and candidate counts.The gains are particularly pronounced on challenging reasoning tasks.
  • Best-of-N selection: +3.1 on GPQA-Diamond and +2.4 on MATH500 at N=16.
  • Rejection sampling: C2 yields +2.0 LC win rate on AlpacaEval 2.0 and +2.2 on Arena-Hard.These results confirm transfer to rejection sampling.

E Inference Latency

C2 incurs substantially higher inference latency than Reasoning RM because it generates rubrics and may retry when rubrics are flagged as misleading. Its rubric analysis also evaluates common generation errors and quality changes.

  • Inference latency: Rubric generation and potential retries when rubrics are flagged as misleading create C2’s latency overhead.
  • Rubric error analysis: C2 raises the error-free rubric rate from 35.0% to 52.5%.Irrelevant criteria, ambiguous rubrics, and missing key constraints decrease, while over-constrained rubrics increase slightly.
  • Rubric error analysis: Irrelevant criteria decrease from 26.3% to 12.5%, and ambiguous rubrics decrease from 33.8% to 5.0%.Missing key constraints also decrease from 21.3% to 10.0%.
  • Rubric error analysis: Over-constrained rubrics increase from 12.5% to 18.8% as the generator pursues more discriminative criteria.The generator occasionally introduces requirements beyond the prompt’s scope.

F.4 Verifier Reasoning Examples

The verifier first analyzes the question and rubric, then evaluates both assistant responses against valid criteria. Examples show that rubric quality and critical verification affect whether judgments follow the prompt’s actual constraints.

  • Verifier limitations: Critical verification can accept helpful rubrics, reject misleading ones, or still make either type of error.These examples demonstrate both the capabilities and limitations of critical verification.
  • Verifier procedure: The verifier formulates an ideal answer and checks whether the rubric aligns with the question before using it.
  • Verifier procedure: Helpful rubrics are used as provided, while misleading rubrics are discarded and replaced with criteria derived from the question.
  • Verifier procedure: The verifier compares both assistant responses against valid criteria and selects the response that better matches the ideal answer.
  • Examples: In the common-elements example, the high-quality rubric enforces the no-extra-data-structures constraint, whereas the low-quality rubric permits contradictory interpretations.
  • Examples: In the plate-and-apple example, the high-quality rubric keeps the apple in the kitchen, while the low-quality rubric incorrectly treats it as concealed underneath the plate.
  • Examples: For the polynomial example, the chosen response identifies roots 1, 2, and 4 and computes k + m = 22.
Loading 2604.13618v1…