Source-linked AI summary
Online Rubrics Elicitation from Pairwise Comparisons
MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, Afra Feyza Akyürek
TL;DR
Static rubrics may miss emerging, sample-specific errors during reinforcement learning with rubric-based rewards. OnlineRubrics elicits criteria from pairwise comparisons of current and control-policy responses; it reports gains of up to 8 percentage points over human-written rubrics on several benchmarks.
Problem
Static rubrics may fail to cover unexpected patterns, including sample-specific behaviors that can fool graders, and may leave correct traits unrewarded.
Source: §1 · p. 1Method
OnlineRubrics dynamically elicits criteria by comparing responses from the current policy and a control model, then augments existing rubrics with those criteria.
Source: §1 · p. 2Results
Up to 8 percentage points of improvement over training exclusively with human-written rubrics are reported on AlpacaEval, GPQA, and Arena-Hard.
Source: §7 · p. 9Takeaways & Limitations
OnlineRubrics aims to surface overlooked errors or emerging desired properties as training progresses across expert and generalist domains.
Source: §7 · p. 9OnlineRubrics tightens the upper bound on the implicit, unmodeled criterion mass rather than fully recovering the true criteria set.
Source: §4.2 · p. 6
Abstract
from arXiv · showhide
Rubrics provide a flexible way to train LLMs on open-ended long-form answers where verifiable rewards are not applicable and human preferences provide coarse signals. Prior work shows that reinforcement learning with rubric-based rewards leads to consistent gains in LLM post-training. Most existing approaches rely on rubrics that remain static over the course of training. Such static rubrics, however, are vulnerable to reward-hacking type behaviors and fail to capture emergent desiderata that arise during training. We introduce Online Rubrics Elicitation (OnlineRubrics), a method that dynamically curates evaluation criteria in an online manner through pairwise comparisons of responses from current and reference policies. This online process enables continuous identification and mitigation of errors as training proceeds. Empirically, this approach yields consistent improvements of up to 8% over training exclusively with static rubrics across AlpacaEval, GPQA, ArenaHard as well as the validation sets of expert questions and rubrics. We qualitatively analyze the elicited criteria and identify prominent themes such as transparency, practicality, organization, and reasoning.
1. Introduction
OnlineRubrics addresses gaps in static rubrics by eliciting criteria from pairwise response comparisons during training. The introduction reports gains over the initial instruct model across multiple benchmarks.
1. Introduction
- Rubric-based rewards use weighted criteria to score long-form responses, extending reinforcement learning to verifiable and non-verifiable aspects.An LLM grader produces binary satisfaction scores for the criteria.Source: §1 · p. 1
- Offline rubrics may miss emerging, sample-specific errors such as self-praising that can fool an LLM-based grader.Existing rubrics may also fail to reward correct traits that they do not cover.Source: §1 · p. 1
- OnlineRubrics elicits criteria through pairwise comparisons of current-policy and control-model responses, then adds them to existing rubrics.The elicited criteria are used with existing criteria to create the policy-gradient reward.Source: §1 · p. 2, Figure 1 · p. 2
- Up to 25% absolute gains over the initial instruct model are reported across GPQA-Diamond, GSM8K, AlpacaEval, and Arena-Hard.The evaluation also includes curated expert and generalist datasets and out-of-distribution public benchmarks.Source: §1 · p. 2
2. Related Work
Related work spans preference-based and verifiable rewards, multi-objective alignment, and rubric-based evaluation and training. OnlineRubrics is positioned as dynamically augmenting rubrics as policies evolve.
Reward Modeling
- Preference-based reward methods use pairwise comparisons [27], while DPO optimizes policies directly on preferences [29] and AI feedback can use fixed principles [4].Other work improves reward models through inference-time scaling [24] or trains judges to reason during evaluation [40].Source: §2 · p. 2
Verifiable Rewards
Multi-Objective Alignment
3. Background
The background describes rubric rewards within policy-gradient training, including GRPO setup and an LLM grader that reduces criterion scores to a scalar reward. The algorithm samples responses from current and control policies before updating the policy.
3. Background
- Rubrics can replace rewards in policy-gradient learning algorithms.Source: §3 · p. 3
3.1 Training Setup
- GRPO uses a policy objective with a probability ratio between target and pre-update policies, and advantages are calculated from normalized rewards.The setup indexes prompts and criteria, samples output groups, and computes rewards independently for each output.Source: §3.1 · p. 3
- The training setup assumes true reward can be modeled from latent criteria and argues that optimal modeling requires eliciting all criteria.Source: §3.1 · p. 3
3.2 Rubric Based Rewards
- An LLM grader scores each response against weighted rubric criteria, and a weighted sum normalized by the maximum possible score yields the scalar reward.The OnlineRubrics algorithm samples prompts, generates current-policy and control-policy responses, and updates the policy by policy gradient.Source: §3.2 · pp. 3–4
4. Online Rubric Elicitation
Static rubrics can miss nuanced errors and emergent desirable qualities, motivating OnlineRubrics, which adds criteria from pairwise response comparisons during training. Its formal motivation relates unmodeled criteria to gradient-estimation error, while treating the method as a step toward—not complete recovery of—the true criteria set.
4. Online Rubric Elicitation
- Initial rubrics may overlook rollout-emergent errors and desirable qualities, including technically accurate but unnecessary information that receives the same score as a focused answer.The section argues that such patterns may only become apparent during rollouts.Source: §4 · p. 4
- OnlineRubrics augments prompt-specific offline criteria with new weighted criteria derived from pairwise candidate-response comparisons.The approach is designed to capture potential errors and useful features.Source: §4 · p. 4
4.1 LLM-based Criteria Elicitation
- Before each policy update, the method compares responses from current and control policies, uses an LLM extractor to identify differences, and adds resulting criteria to the rubric.The control responses may come from a reference or previous policy.Source: §4.1 · pp. 4–5
- Criteria elicitation first identifies meaningful response differences and then removes duplicate or substantially overlapping criteria.By default, eight rollout pairs are compared and about eight criteria are extracted.Source: §4.1 · p. 5
- The previous-policy control variant performs strongly compared with, and may perform better than, the reference-policy control variant.The passage reports this empirical observation with reference to Table 2.Source: §4.1 · p. 5
4.2 A Formal Motivation for OnlineRubrics
- Under GRPO-style updates, the gradient difference is bounded by the implicit criteria-weight norm and the expected squared policy-score norm; adding criteria can improve estimation of the true gradient.The formal argument links closer approximation of the true criterion set with improved training stability and sample efficiency.Source: §4.2 · p. 5
- OnlineRubrics tightens an upper bound on implicit, unmodeled reward mass but does not completely recover the true criteria set.The formal setup distinguishes criteria represented in the rubric from implicit criteria outside it.Source: §4.2 · pp. 5–6
5. Datasets
The paper uses Generalist and Expert Rubrics datasets, both built around human-authored, weighted, binary-checkable criteria. Evaluation uses held-out dataset subsets and public benchmarks.
5. Datasets
- Generalist Rubrics contains consented, curated real-world single-turn prompts with human-authored prompt-specific weighted criteria; Expert Rubrics covers Physics, Chemistry, Biology, and Math problem sets with expert rubrics and sample ratings.Expert tasks bundle a prompt, rubric, sample model responses, and detailed rubric ratings.Source: §5 · p. 6
- Across both datasets, human-written criteria follow principles of mutual exclusivity, atomicity, objectivity, and self-containment for reliable verification and dense rewards.Source: §5 · p. 6
- Held-out subsets are used for evaluation, with rubric scores and win rates judged by Gemini 2.5 Pro, alongside public-benchmark evaluation on GPQA-Diamond, GSM8K, AlpacaEval, and Arena-Hard.The dataset statistics are presented in Table 1.Source: §5 · p. 6, Table 1 · p. 6
6. Experiments and Results
Experiments evaluate grader selection, baseline comparisons, and online elicitation during training. OnlineRubrics improves results across evaluation settings, while elicited criteria add new evaluative dimensions over time.
6.3 Results and Discussion
- On fixed Generalist and Expert evaluation sets without elicited rubrics, both OnlineRubrics methods outperform Offline Rubrics (Human) and Likert-scale LLM-judge scores.Training with rubrics also scores higher and is more sample-efficient than training with LLM-Judge scores.Source: Figure 4 · p. 7, §6.3 · p. 8
6. Experiments and Results
- Training uses Qwen-2.5-7B-Instruct with GRPO for 3 epochs, evaluating each dataset’s evaluation set 10 times per epoch.The extractor is o3-mini, and the number of pairwise comparisons is 8.Source: §6 · p. 7
6.1 Verifier Selection
- Human judgments cover 500 prompts per dataset, and GPT-4.1-mini is selected as default grader for balancing alignment with grades and inference costs.The grader selection follows evaluation of several LLM graders; rubrics training grades multiple rollouts per prompt.Source: §6.1 · p. 7
6.3 Results and Discussion
- On AlpacaEval, adding elicited rubrics to human-written offline rubrics through OnlineRubrics-πref raises win rate from 46.4% to 55.0% and length-controlled win rate from 28.0% to 31.5%.Source: §6.3 · p. 9
6.2 Baselines
- Across the benchmarks evaluated, using human-written rubrics is often significantly better than using synthetic rubrics, which were created with o3-mini using prompts from the Generalist and Expert Rubrics datasets.Source: §6.2 · pp. 7–8
6.3 Results and Discussion
- OnlineRubrics consistently outperforms Universal Requirements across benchmarks and surpasses pointwise elicitation, which the authors say lacks pairwise differences between better and worse responses.Pointwise elicitation scores 48.1 versus 54.0 and 55.0 on AlpacaEval, and 51.1 versus 55.7 and 56.5 on Arena-Hard.Source: §6.2 · p. 8, §6.3 · p. 9
6.4 Qualitative Analysis
- Elicited criteria add evidence grounding, reproducibility, anti-gaming, practicality, resource awareness, structural organization, causal reasoning, and uncertainty handling.The analysis finds that online elicitation tends to expand and strengthen rubrics over time as new errors or weaknesses are exposed.Source: §6.4 · p. 9
7. Conclusion
OnlineRubrics dynamically elicits criteria from pairwise response comparisons during reinforcement learning. It reports gains of up to 8 percentage points over human-written rubrics on AlpacaEval, GPQA, and Arena-Hard.
7. Conclusion
- OnlineRubrics dynamically elicits new criteria from pairwise comparisons of responses during reinforcement learning.The approach aims to surface overlooked errors or emergent desired properties as training progresses.Source: §7 · p. 9
- Up to 8 percentage points of improvement over human-written rubrics are reported on AlpacaEval, GPQA, and Arena-Hard.The conclusion characterizes the gains as robust across expert and generalist domains.Source: §7 · p. 9
- OnlineRubrics adapts as training evolves, capturing emergent behaviors and strengthening alignment beyond fixed rubrics.Source: §7 · p. 9
A. Proof for Proposition 1
When E(Y)=0, centering Y does not change the expectation.
- When E(Y)=0, centering Y leaves the expectation unchanged.Source: A. Proof for Proposition 1 · p. 13
B. Data Samples
The data samples show rollouts from current and reference policies alongside human and elicited rubrics, whose criteria have importance weights.
- Samples pair rollouts from current and reference policies with human and elicited rubrics.Source: B. Data Samples · p. 13
- Generalist criteria are weighted from 1–5, while Expert criteria use weights from −10 to 10.Source: B. Data Samples · p. 13
C.1 Training Settings.
OnlineRubrics is trained for three epochs on the Generalist and Expert Rubrics training sets using GRPO, with 16 rollouts per sample.
- Training runs for three epochs on the Generalist and Expert Rubrics training sets, following GRPO with 16 rollouts per sample.Source: §C.1 · p. 13
- GPT-4.1-mini serves as LLMgrader and o3-mini as LLMextractor, with eight pairwise comparisons per instance.Source: §C.1 · p. 13
- Optimization uses a 5e−6 learning rate, 0.1 warmup ratio, and 0.01 KL-divergence coefficient on eight H100 GPUs.Per-device batch size is 6 with 2 gradient-accumulation steps, for an effective batch size of 96.Source: §C.1 · p. 13
C.2 Evaluation Settings.
The evaluation settings specify how each benchmark and rubric-set comparison is scored. The appendices also provide system-prompt templates and summarize the most frequent clusters of elicited rubric refinements.
C.2 Evaluation Settings.
- Rubric-set evaluation uses GPT-4.1-mini for scores and Gemini-2.5-Pro for win judgments, with response order flipped for a second judgment.AlpacaEval reports win rate and length-controlled win rate; Arena-Hard reports win rate, GPQA-Diamond average accuracy across four runs, and GSM8K strict-match accuracy.Source: §C.2 · p. 13
D. System Prompt Templates
- The appendix supplies templates for extracting and de-duplicating criteria, grading responses, judging scores and win rates, and generating synthetic offline rubrics.Source: D. System Prompt Templates · p. 14
E. Qualitative Rubric Clusters
- The qualitative analysis reports refinement clusters by name, description, and share of samples, sorted by proportion.Source: E. Qualitative Rubric Clusters · p. 14, Figure 7 · p. 16