PaperlayerView exact revision on arXiv

Source-linked AI summary

Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

Ruipeng Jia, Yunyi Yang, Wen Wang, Yuxin Wu, Yongbo Gai, Siyuan Tao, Mengyu Zhou, Jianhe Lin, Xiaoxi Jiang, Guanjun Jiang

arXiv:2602.14069v3cs.CL

TL;DR

Scalar reward models bottleneck open-ended alignment by compressing multidimensional preferences into opaque scores, while many tasks lack verifiable ground truth. OpenRS replaces this with explicit meta-rubrics, adaptive pairwise criteria, and pointwise verifiers, achieving stronger reward-model and RL results across reported evaluations. The approach still incurs substantial cost and relies on human-in-the-loop domain adaptation.

  • Problem

    Scalar reward models compress multidimensional preferences into opaque scores, while non-verifiable tasks lack ground-truth signals for reliable reward specification.

  • Method

    OpenRS uses hierarchical meta-rubrics, difference-grounded pairwise adaptive rubrics, criterion-wise external aggregation, and pointwise verifiers without judge-specific training.

  • Results

    +5.1 average points over the strongest open scalar reward model, 89.4 vs. 84.3, while RL evaluations average 71.3 vs. 68.4.

  • Takeaways & Limitations

    Explicit, inspectable rubric reasoning can serve as a scalable reward interface for open-ended RL while retaining verifiable constraints for objective subtasks.

  • Takeaways & Limitations

    Domain meta-rubric adaptation uses human-in-the-loop refinement because repeating automated refinement for every domain is prohibitively expensive.

Abstract

from arXiv · show

Scalar reward models compress multi-dimensional human preferences into a single opaque score, creating an information bottleneck that often leads to brittleness and reward hacking in open-ended alignment. We argue that robust alignment for non-verifiable tasks is fundamentally a principle generalization problem: reward should not be a learned function internalized into a judge, but an explicit reasoning process executed under inspectable principles. To operationalize this view, we present the Open Rubric System (OpenRS), a plug-and-play, rubrics-based LLM-as-a-Judge framework built around Pairwise Adaptive Meta-Rubrics (PAMR) and lightweight Pointwise Verifiable Rubrics (PVRs), which provide both hard-constraint guardrails and verifiable reward components when ground-truth or programmatic checks are available. OpenRS uses an explicit meta-rubric -- a constitution-like specification that governs how rubrics are instantiated, weighted, and enforced -- and instantiates adaptive rubrics on the fly by conditioning on the semantic differences between two candidate responses. It then performs criterion-wise pairwise comparisons and aggregates criterion-level preferences externally, avoiding pointwise weighted scalarization while improving discriminability in open-ended settings. To keep principles consistent yet editable across various domains, we introduce a two-level meta-rubric refinement pipeline (automated evolutionary refinement for general principles and a reproducible human-in-the-loop procedure for domain principles), complemented with pointwise verifiable rubrics that act as both guardrails against degenerate behaviors and a source of verifiable reward for objective sub-tasks. Finally, we instantiate OpenRS as reward supervision in pairwise RL training.

1. Introduction

Open-ended alignment remains difficult because scalar reward models compress multi-dimensional preferences into opaque scores, while objective verification is unavailable for many tasks. OpenRS addresses this by using explicit principles, adaptive pairwise rubrics, and verifiable components as reward supervision.

  • Motivation: Scalar reward models compress rich human preferences into one score and can learn shortcuts that enable reward hacking.These models may become miscalibrated and exploit spurious cues instead of improving response quality.
  • Motivation: Open-ended tasks lack explicit ground truth, while desired behavior spans multiple potentially competing dimensions.Verifiable rewards scale well for correctness or formatting, but their scope is limited when no reference answer exists.
  • OpenRS: OpenRS reframes robust reward supervision as principle generalization, using explicit and auditable meta-rubrics instead of learned scalar functions.The meta-rubric specifies rubric generation, criterion priorities, and non-negotiable failure modes.
  • OpenRS: PAMR conditions rubric generation on semantic differences between two responses, then performs criterion-wise pairwise comparisons.The system combines adaptive subjective evaluation with lightweight pointwise verifiable rubrics for objective checks.
  • Contributions: Across four reward-modeling benchmarks, OpenRS outperforms prior scalar and trained generative judges, while pairwise RL training yields consistent policy gains.The contribution passage reports both benchmark superiority and improvements on public and industrial policy evaluations.

2. Preliminary

GRPO normalizes rewards within sampled output groups and constrains policy updates, but exhaustive pairwise evaluation scales quadratically. BRPO addresses this cost by comparing each response with one dynamically sampled reference anchor.

  • GRPO: GRPO eliminates the value-function critic by using group-relative reward normalization for advantage estimation.A behavior policy samples a group of outputs, whose rewards determine relative advantages.
  • GRPO: GRPO constrains policy updates within a trust region using clipping, supporting training stability and sample efficiency.The objective uses a clipping mechanism analogous to PPO.
  • Pairwise feedback: Naive pairwise evaluation of a group of size G requires O(G^2) comparisons.This quadratic cost makes direct pairwise feedback difficult to integrate with group-based RL.
  • BRPO: BRPO samples one response as a dynamic anchor and evaluates every other response relative to it.This bootstrapping strategy reduces GenRM calls from O(G^2) to O(G).

3. Approach

OpenRS combines hierarchical meta-rubrics, difference-grounded pairwise adaptive rubrics, and pointwise verifiers into a transparent reward interface. Its general principles are refined by automated search, while domain-specific principles are adapted with human feedback.

  • Overall Framework: OpenRS uses a meta-rubric to specify evaluation criteria, weights, and failure modes, while adaptive rubrics and verifiers produce the reward signal.The system is designed for structured evaluation and scalable reinforcement learning in general domains.
  • Overall Framework: The reward decomposes into pairwise adaptive judgment for subjective dimensions and pointwise verifiable supervision for objective dimensions.This separates relative preference assessment from absolute checks such as correctness or formatting.
  • Hierarchical Meta Rubric: OpenRS composes a general meta-rubric with domain extensions, keeping the judge untrained and fixed during evaluation.General principles are captured by Mgen, domain-specific criteria by Mdom, and adaptation occurs at the principle level.
  • Pairwise Adaptive Rubric: For each response pair, OpenRS extracts semantic differences and conditions the meta-rubric to create a focused, weighted criterion set.The adaptive rubric is specific to the comparison rather than static across all response pairs.
  • Pairwise Adaptive Rubric: Criterion-level comparative verdicts are externally aggregated into a weighted pairwise score, avoiding the need for an absolute quality score.Each criterion receives a comparative value, and the resulting judgments remain inspectable.
  • Pointwise Verifiable Rubric: Pointwise verifiers produce deterministic signals for single-output constraints such as word count, format, or strict instruction following.These checks provide objective reward components and hard-constraint guardrails.
  • Meta-Rubric Refinement: Meta-rubric refinement uses evolutionary beam-style search for general principles and human-in-the-loop adaptation for domain principles.The general refinement is treated as discrete optimization over criterion-level edits using evaluation feedback.
  • Meta-Rubric Refinement: Asymmetric GRPO updates the refinement policy only through top-performing rollouts, filtering failures under sparse, skewed rewards.This concentrates learning on successful rubric edits and is intended to stabilize iterative improvement.

17 end

OpenRS integrates adaptive pairwise and verifiable rubric scores into reinforcement learning and shows stronger benchmark alignment after rubric refinement. Its domain adaptation remains costly because human-in-the-loop refinement is used where automated refinement is expensive.

  • Rubric Refinement: Replacing refined Mgen with the seed rubric reduces RewardBench v2 performance from 90.7 to 85.4.The ablation attributes the difference to stronger evaluation principles discovered through evolutionary refinement.
  • Domain Adaptation: Domain-specific adaptation is most useful for heterogeneous subjective preferences, whereas objective-norm benchmarks can be well covered by Mgen alone.The cited discussion contrasts domains such as PPE Chinese with benchmarks governed by stronger objective norms.
  • Benchmark Comparison: Table 1 compares OpenRS with scalar reward models, generative judges, and rubric-based methods, marking backbone, best scores, reimplementations, and unavailable results.The caption defines parentheses, bold values, asterisks, and dash entries used in the comparison.
  • Domain Adaptation: Human-in-the-loop domain adaptation avoids repeating the costly general refinement training for every domain.This reduces repeated compute but introduces expert labor and a mixed automated/manual refinement paradigm.
  • RL Integration: OpenRS combines pairwise adaptive and pointwise verifiable scores to provide subjective preference feedback and hard-constraint enforcement in RL.The pointwise component aggregates verifiable constraint signals, while the adaptive component supplies criterion-weighted preferences.

4. Experiments

OpenRS achieves state-of-the-art alignment with human preferences across four benchmarks, while its rubric-based design remains effective across judge backbones and compute scales. The strongest configuration improves average performance over the strongest open scalar reward model and offers a favorable accuracy–compute trade-off.

  • Main results: OpenRS achieves state-of-the-art alignment across RM-Bench, JudgeBench, RewardBench v2, and PPE Preference, with the best overall average.It outperforms the strongest open scalar reward model by +5.1 points on average, reaching 89.4 versus 84.3.
  • Benchmark results: 93.0 on RM-Bench, 93.3 on JudgeBench, 90.7 on RewardBench v2, and 82.3 on PPE Preference exceed the strongest reported baselines on each benchmark.The gains are +0.2, +11.3, +6.6, and +2.7 points, respectively.
  • Foundation-model scaling: OpenRS performance scales with judge-backbone strength, improving from 84.9 with Qwen3-30B-A3B to 89.4 with Qwen3-235B-A22B on average.Non-Qwen backbones also surpass strong scalar reward-model baselines on key benchmarks, supporting applicability beyond one model family.
  • Foundation-model scaling: Even with an unmodified Qwen3-8B judge, OpenRS reaches 92.4 on RM-Bench and 89.5 on JudgeBench, essentially matching the 235B configuration on those benchmarks.Against the identical-backbone Skywork-Reward-V2-Qwen3-8B, OpenRS averages 82.6 versus 77.6 across all four benchmarks without judge training.
  • Foundation-model scaling: Qwen3-30B-A3B reaches 84.9 on average, exceeding Skywork-Reward-V2-Llama-3.1-8B at 84.3.The comparison uses only 3B active parameters for Qwen3-30B-A3B.
  • Compute–quality trade-off: Qwen3-Instruct models form a strong Pareto frontier, with Qwen3-235B-A22B combining the best overall evaluation quality with lower inference compute than heavier backbones.This trade-off motivates selecting Qwen3-235B-A22B-Instruct-2507 as the default judge backbone for subsequent RL training.

5. Analysis

The analysis finds that optimized principles, pairwise comparison, and difference-grounded criteria are central to OpenRS performance. It also examines refinement-policy stability, RL integration, and the possibility of qualitative behavioral changes in general-domain training.

  • Refine the meta rubric: 85.4 vs. 90.7 on RewardBench v2 and 74.6 vs. 82.3 on PPE Pref/ZH shows that refined principles, rather than an identical prompt scaffold, carry evaluation quality.The comparison replaces the refined meta-rubric with the seed rubric while keeping the evaluation prompt byte-identical.
  • Pairwise over Pointwise: Pairwise OpenRS outperforms its pointwise variant, supporting pairwise comparison as the recommended default for nuanced preferences.The pointwise variant remains useful when pairwise comparisons are infeasible.
  • Difference-grounded criteria: 86.2/75.5 and 88.9/80.6 for query-only and no-difference-mechanism ablations show that explicit semantic-difference analysis improves criterion selection.The results indicate that structured difference analysis contributes beyond merely exposing the judge to both responses.
  • Refinement-policy training: Frozen refinement quickly plateaus, standard GRPO is volatile, and Asym-GRPO produces more stable, sustained improvements in Figure 3.Asym-GRPO reinforces successful mutations while masking gradients from rollouts outside the top group.
  • General RL: OpenRS replaces scalar reward scoring in controlled RL comparisons while tracking entropy and “Same” judgments as diagnostics of exploration and judge decisiveness.The supplied passages describe fixed policy and data settings, asynchronous throughput, and additional appendix analyses, but do not report the corresponding Table 3 or Table 4 values.
  • Aha Moment in General Domains: The analysis frames OpenRS as a soft rule-based verifier that preserves criterion-level comparative evidence before forming RL advantages.This design is presented as a route toward richer open-ended reward signals than learned pointwise scalar proxies.
  • Aha Moment in General Domains: Case-study passages report stronger subjective consciousness, richer emotional expression, distinct personality, and more willingness to express stances than scalar-RM-trained responses.The paper presents this qualitative shift as a possible general-domain analogue of an RL “Aha Moment.”

6. Related Work

Related work contrasts scalar reward models, direct LLM judges, rubric-based rewards, and pairwise methods. OpenRS differs by generating adaptive rubrics from response differences while keeping principles stable, explicit, and externally aggregated.

  • Scalar reward models: Scalar reward models provide strong baselines but compress heterogeneous preferences into one number, obscuring which criterion drove a judgment.The related work includes Skywork-Reward, ArmoRM, HelpSteer2/Nemotron, and INF-ORM as examples.
  • LLM-as-a-Judge and rubric evaluation: Direct LLM-as-a-Judge evaluation scales assessment but remains vulnerable to positional, verbosity, and sycophancy-style biases.Automated rubric construction seeks to improve structure and scalability beyond expert-authored rubrics.
  • Rubric-based evaluation: Earlier rubric-as-reward systems often map criterion satisfaction into weighted or implicit scalar rewards, limiting the preservation of comparative evidence.Causal-rubric work separately targets spurious cues by distinguishing causal quality attributes from neutral augmentations.
  • OpenRS distinction: OpenRS instantiates adaptive rubrics from semantic differences at inference time, while its meta-rubric remains a stable, inspectable guideline.It declares criteria and weights before scoring, produces per-criterion verdicts, and aggregates preferences externally.
  • From RLVR to open-ended alignment: OpenRS extends verifiable-reward ideas toward open-ended alignment by combining adaptive pairwise rubrics with optional verifiable constraints.The related work identifies pointwise scalarization as vulnerable to discriminability limits and reward gaming.

7. Conclusion

The conclusion presents OpenRS as principle-based reward supervision that replaces opaque scalarization with explicit, editable rubric reasoning. Across reward-modeling and RL evaluations, it reports stronger alignment and feasible deployment, while identifying efficiency and broader validation as future priorities.

  • Contribution: OpenRS operationalizes principle generalization through PAMR, PVRs, difference-grounded criteria, criterion-wise pairwise comparisons, and external preference aggregation.The system is designed as a modular rubric-based reward interface for non-verifiable tasks.
  • Reward-modeling results: +5.1 overall average, 89.4 vs. 84.3, is reported over the strongest open scalar reward model across four reward-modeling benchmarks.An unmodified Qwen3-8B judge also exceeds a size-matched trained scalar reward model by +5.0.
  • RL results: 71.3 vs. 68.4 average on public evaluations is reported when OpenRS replaces scalar reward models in RL, with asynchronous batching preserving throughput feasibility.The conclusion characterizes these as consistent gains under the reported deployment setup.
  • Future directions: Future work targets judge-cost reduction, adversarial and prompt-injection robustness, rubric-drift monitoring, automated domain adaptation, agent settings, and broader RL validation.The stated validation scope includes more policy sizes, languages, and real-world tasks.

8. Limitations

The paper’s main scope boundary is that domain adaptation currently relies on a costly human-in-the-loop workflow rather than the automated refinement used for general principles.

  • Domain adaptation cost: Domain adaptation is prohibitively expensive to repeat because general meta-rubric refinement requires training a refinement policy.The authors therefore use human-in-the-loop domain adaptation, introducing expert labor and a mismatch between automated and manual refinement.
  • Residual judge bias: The LLM may still form implicit preferences when both responses are visible, although difference-first design and external aggregation mitigate rather than eliminate this risk.The limitation concerns residual judge subjectivity in the pairwise rubric pipeline.
  • Evaluation scope: The end-to-end RL comparison isolates reward quality with a single policy model, leaving broader deployment validation across policy sizes, languages, and settings for future work.The supplied passage truncates the final list of broader validation settings.

A.1. Detailed Performance Comparison

OpenRS is compared across fine-grained benchmark dimensions, using adaptive pairwise evaluation and bidirectional consistency checks to address weaknesses of scalar reward models. Its deployment guidance distinguishes objective tasks, where general principles suffice, from open-ended domains requiring domain adaptation.

  • Fine-grained benchmark analysis: Scalar reward models often degrade on hard RM-Bench examples and show imbalanced performance across RewardBench v2 categories.The analyses contrast difficulty levels and categories to expose weaknesses that aggregate scores can conceal.
  • Adaptive evaluation: OpenRS dynamically generates context-aware criteria so each response pair is evaluated on the quality dimensions most relevant to that comparison.This addresses the stated limitation that monolithic scalar scoring cannot adaptively emphasize different evaluation dimensions.
  • Arena-like evaluation: The in-house QA and Writing evaluation uses 1,000 production-log queries per task with human-annotated candidate-response preferences, but its results remain ongoing.The setup targets authentic user needs and subjective quality judgments; the authors defer conclusions until complete results are available.
  • Training-time behavior: During RL training, the Same rate rises from approximately 0.3 to 0.45 by step 70 and then plateaus, while similar-policy rollouts produce more Same judgments than benchmark pairs.A query is filtered only when indistinguishable pairs exceed a threshold, so the pair-level Same rate is higher than the query-level drop rate.
  • Evaluation procedure: OpenRS uses bidirectional pairwise evaluations, declaring a winner only when forward and reverse judgments agree and otherwise treating the comparison as a tie.This procedure is described for JudgeBench and PPE; RewardBench v2 aggregates one chosen response against multiple rejected responses using the same bidirectional comparisons.
  • Domain generalization: General principles suffice for RM-Bench, JudgeBench, and RewardBench v2, whereas PPE Preference’s subjective, domain-dependent categories motivate domain adaptation.The paper recommends Mgen for clear quality standards and Mdom adaptation for open-ended user-facing scenarios with diverse preferences.

C. Case Study

The case study presents several real-world user scenarios from end-to-end reinforcement-learning training with OpenRS. The supplied passages identify the cases and language variants but do not report their outcomes.

  • C. Case Study: OpenRS is used to train an end-to-end reinforcement-learning model, followed by case studies from real-world user scenarios.The section introduces five cases, including Chinese and English versions of Case 1.
  • Case coverage: The supplied case labels cover Case 1 through Case 5, with separate Chinese and English figures for Case 1.The passages provide figure identifiers but no textual comparison or outcome for the cases.
Loading 2602.14069v3…