PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR

Utkarsh Tyagi, Xingang Guo, MohammadHossein Rezaei, Daniel George, Anas Mahmoud, Jackson Lee, Bing Liu, Yunzhong He

arXiv:2605.20164v1cs.AI

TL;DR

Rubric RL must optimize multiple qualitative criteria, but static weights can misalign human importance with what currently teaches the policy. POW3R preserves the rubric target while reallocating training pressure using rollout contrast, beating baselines in 24 of 30 comparisons and matching them in 2.5–4× fewer steps.

  • Problem

    Rubric rewards expose multidimensional quality, but static aggregation can direct training pressure toward saturated or unreachable criteria rather than currently learnable ones.

  • Method

    POW3R preserves human weights and category balance while adapting within-category criterion weights toward rollout-level contrastiveness during GRPO training.

  • Results

    POW3R beats binary, static-scalar, and category-balanced rewards on 24 of 30 comparisons and matches them in 2.5–4× fewer training steps.

  • Takeaways & Limitations

    Rubric aggregation can be treated as a training-time design choice that extracts more optimization signal without changing the evaluation target.

  • Takeaways & Limitations

    POW3R relies on LLM judges for both training rewards and evaluation, so judge biases can propagate into training-pressure allocation and reported scores.

Abstract

from arXiv · show

Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion's human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy's outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins $24$ of $30$ base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in $2.5$--$4\times$ fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.

1 Introduction

Rubric RL must balance human-defined evaluation priorities with the criteria that currently provide learning signal. POW3R preserves the rubric objective while reallocating within-category pressure toward criteria that distinguish rollouts.

  • Motivation: Many target behaviors are inherently multidimensional, requiring answers to satisfy factuality, completeness, evidence fidelity, formatting, and instruction-following simultaneously.Rubric grading exposes structure that exact-answer scoring can miss, supporting RL post-training beyond strictly verifiable domains.
  • Problem: Static weighted sums conflate a criterion’s desired final-answer importance with its current usefulness as a training signal.Criteria passed by every rollout or by none contribute the same reward to the group and therefore cancel from GRPO advantages.
  • Diagnostic: Roughly half of rubric criteria are non-contrastive for a fresh policy: 17–26% are saturated and 20–33% are dead across both policies and settings.Only the remaining criteria can produce a contrastive gradient for group-relative learning.
  • Diagnostic: Static aggregation routes 45–51% of within-category training pressure to criteria that cannot move the policy.Human weight and rollout variance are essentially decoupled, so high-importance criteria are not necessarily informative for optimization.
  • Contribution: POW3R preserves human weights and category balance while reallocating within-category pressure toward criteria that currently distinguish rollouts.The framework treats the evaluation rubric as the target while adapting criterion-level reward weights during training.

2 Related work

Prior work extends reinforcement-learning post-training beyond deterministic verifiers through rubric-based rewards, multi-objective formulations, and multimodal RLVR. These approaches address heterogeneous quality criteria but raise challenges in rubric construction and multi-reward optimization.

  • Rubric-based rewards and the policy-aware view: Rubric-based rewards decompose response quality into prompt-specific criteria scored by an LLM judge, extending RL post-training beyond deterministic verifiers.Expert-written benchmarks have scaled this signal in medicine and multi-turn spoken dialogue, while synthetic or semi-automatic pipelines reduce rubric-authoring cost.
  • Multi-reward RL and multimodal RLVR: Multi-reward RL treats alignment as multi-objective rather than scalar optimization, complementing RLHF/RLAIF methods that compress rich feedback into a single reward target.Scalarization can hide heterogeneous values and failure modes.
  • Multi-reward RL and multimodal RLVR: Liu et al. show that naively normalizing multi-reward rollouts under GRPO collapses distinct reward combinations into identical advantages.This result highlights a limitation of directly applying scalar advantage normalization to multi-reward settings.

3 Preliminaries

GRPO trains policies from standardized outcome rewards over groups of sampled outputs, while rubric rewards score prompt-specific criteria using static human-weighted sums. These preliminaries motivate adapting training usefulness without changing criterion scores or human weights.

  • GRPO: GRPO samples groups of G outputs from the old policy and optimizes the policy using per-token probability ratios.The per-token ratio is ri,t(θ) = πθ(oi,t | q, oi,<t)/πθold(oi,t | q, oi,<t).
  • GRPO: Outcome supervision assigns each output a scalar reward and standardizes rewards within its group to produce token-level advantages.When all rollouts tie, the standardized advantage is zero, so that group contributes no gradient for the step.
  • Rubric rewards: Rubric rewards decompose response quality into prompt-specific criteria scored by an LLM judge, with each criterion assigned a static human weight and category label.Criterion scores lie in [0, 1], with a score of 1 indicating that the response satisfies the criterion.
  • Rubric rewards: The standard rubric reward is a static weighted sum that assumes balanced categories, similarly informative within-category criteria, and weights that encode both importance and training usefulness.The proposed framework relaxes these assumptions while leaving criterion scores and human weights unchanged.

4 Method

POW3R preserves the rubric, judge scores, human weights, and category balance while reallocating within-category reward pressure toward criteria that distinguish current rollouts. It smooths, normalizes, bounds, and exponentially averages these adaptive factors before feeding rewards into standard GRPO without changing the optimizer.

  • Reward aggregation: POW3R changes only reward aggregation before GRPO standardization, keeping the rubric, judge scores, and human weights fixed.Reallocation occurs within categories toward criteria that distinguish the current rollout group.
  • Rollout contrast: POW3R computes criterion pass-rate and variance signals across G rollouts, retaining prior factors when fewer than ⌈0.75G⌉ valid verdicts are available.The valid-verdict set determines whether a criterion has enough judgments for updating.
  • Adaptive weighting: POW3R smooths variance, category-normalizes it, blends toward 1, clips factors, and updates them with an exponential moving average.λ controls prior-versus-contrast blending, βema controls response speed, and [αmin, αmax] bounds deviation from 1.
  • GRPO integration: The resulting weights preserve uniform category mass, use the human weights as a prior, and are applied to all G rollouts before GRPO.The method requires no optimizer change.

5 Experimental setup

The experiments span multimodal and text-only rubric-RL settings, using multiple base policies, criterion-level judging, and fixed-target held-out evaluation. POW3R is compared against binary, static scalar, category-balanced, and no-RL references, with transfer benchmarks for MM.

  • Datasets: The study uses HealthBench’s 500-task hard test split plus a separate 10% English dev slice, alongside a 10k-task multimodal dataset.HealthBench uses physician-authored point-valued criteria; MM exposes criterion-level categories and static importance weights.
  • Models: Experiments post-train three vision-language policies on MM and three text-only policies on HealthBench, with additional diagnostics spanning Qwen3-VL-4B-Instruct and Gemma 3 12B-IT.The diagnostic tests whether findings are specific to a single base model.
  • Reward judging: Each prompt–rollout–criterion triple receives an independent reasoning-then-verdict judgment, with GPT-5.4-nano for training and GPT-5.4-mini for held-out evaluation.Judgments return a one-sentence rationale and binary {0, 1} verdict for aggregation, reducing judge–training entanglement.
  • Baselines: The comparison includes no RL, binary, static scalar, category-balanced, and POW3R dynamic rewards, all using the same rubric set and judge.The binary baseline is all-or-nothing, while static scalar and category-balanced rewards provide fixed aggregation references for POW3R’s dynamic reward.
  • Evaluation: Held-out evaluation reports mean rubric reward on 1,000-task MM and 500-task HealthBench tests, with MM’s normalized static weighted target fixed across reward constructions.MM-trained policies are additionally evaluated on HallusionBench, POPE, MM-IFE, MMVetV2, MathVista, and RealWorldQA.

6 Results and analysis

POW3R is the strongest reward construction across the main rubric objectives, consistently improving both average rubric quality and strict all-criteria success. Its rollout-aware weighting also concentrates optimization on categories with learnable disagreement, producing faster gains under the same GRPO setup.

  • Main results: 24 of 30 base-policy/metric comparisons favor POW3R, including every MM rubric-reward and strict-completion column and every HealthBench overall-reward column.The six exceptions are concentrated in external VLM benchmarks and HealthBench strict perfect-score evaluation.
  • Per-category analysis: POW3R leads on every analyzed MM rubric category, with the largest gains on Visual Perception, Visual Reasoning, Truthfulness, Content, and Instruction Following.Writing Style changes least because most of its criteria are already passed by the base policy, leaving little rollout disagreement to exploit.
  • Main results: POW3R Pareto-dominates the other reward constructions on MM test rubric reward and strict completion for every base policy.Strict completion distinguishes complete satisfaction of every required criterion from partial-credit improvements in mean rubric score.
  • Cross-setting consistency: Across all six setting/base-policy combinations, the gain ordering is Rdyn > Rcat > Rscalar > 0, with POW3R’s smallest gain still +3.7 pp.The ordering remains unchanged across multimodal and text-only settings.
  • Training efficiency: POW3R separates from static aggregations by the first logged checkpoint and remains higher through step 501 under matched compute.All methods use the same GRPO recipe, prompt budget, optimizer schedule, and evaluation checkpointing; category-balanced reward requires 332 steps to cross the cited threshold, while only POW3R crosses 50.0 within the schedule.

7 Limitations

POW3R’s limitations include dependence on LLM-judge verdicts and evaluation on a narrow set of rubric datasets. Its policy-aware reweighting also tracks rollout contrast, whose signal varies across categories.

  • Policy-aware signal: Categories with more rollout contrast receive greater POW3R reweighting, so the framework’s signal depends on category-specific variance during training.Figure 7 compares mean rollout variance with the within-category standard deviation of POW3R’s factors α_j over the full MM-train schedule.
  • Reliance on LLM judges: POW3R depends on LLM judges for both training rewards and reported scores, so judge, prompt-template, or sampling choices can shift exact results.Training uses GPT-5.4-nano and held-out evaluation uses GPT-5.4-mini, calibrated against a reference judge with 91–94% verdict agreement.
  • Scarcity of rubric datasets with static human weights: Public rubric-RL datasets combining prompt-specific criteria, integer human weights, multiple categories, and held-out scale are rare, limiting domain coverage.The study therefore authors MM in-house and uses HealthBench, leaving domains such as long-form coding feedback, scientific writing, and multilingual instruction following untested.

8 Conclusion and future work · Appendix · Appendix index.

POW3R separates final-answer priorities from criteria that can currently teach the policy, using rollout-level contrast to improve rubric optimization without changing the rubric target. The appendices document datasets, judge selection, prompts, training settings, rubric categories, and qualitative examples.

  • 8 Conclusion and future work: POW3R separates what should matter in the final answer from what can currently teach the policy.Its policy-aware reward redirects training pressure without changing the rubric target.
  • 8 Conclusion and future work: POW3R yields higher rubric reward, higher strict completion, cleaner two-objective dominance, and faster training across MM, HealthBench, and two model families.
  • Appendix index.: Appendix A documents rubric annotations, contributor demographics, and split details for MM.
  • Appendix index.: Appendix B covers reference-judge calibration, cost–quality tradeoffs, and operating points, while Appendix C provides rubric-judge prompts for training and evaluation.
  • Appendix index.: Appendix D consolidates GRPO and POW3R training hyperparameters shared across all runs, and Appendix E lists six MM rubric categories with representative criteria from MM-train.
  • Appendix index.: Appendix F presents four MM held-out tasks with side-by-side outputs from the base policy, static baselines, and POW3R.

A Dataset and contributor details … A.4 Signed HealthBench criteria during training and evaluation

The appendix details the multimodal dataset’s annotation, contributor, quality-control, and split procedures, then explains how signed HealthBench criteria are transformed for POW3R training while evaluation retains original scoring. These design choices define the released data and ensure POW3R’s reward calculations are well-defined.

  • A Dataset and contributor details: The data-collection appendix provides details on contributor demographics, collection procedures, and multimodal dataset splits.It frames these details as supplementary information about the multimodal dataset.
  • A.1 Rubric annotations and weights: The multimodal dataset includes prompt-specific rubric criteria annotated by category, explicitness, subjectivity, and static human weight.Explicit criteria state prompt or image-grounded requirements, while implicit criteria capture additional high-quality behaviors such as uncertainty explanations.
  • A.2 Contributor demographics and selection: The multimodal dataset was authored by vetted English-proficient contributors experienced in analytical writing, visual reasoning, and rubric-based evaluation.Fine-grained contributor demographics, collection dates, geographic distribution, and job-title breakdowns are omitted from the release.
  • A.3 Dataset funnel and splits: The quality-control funnel discarded examples failing independent review or dataset-level criteria before the final 10,000 examples were split into training, development, and test sets.The passage describes this process as strict filtering.
  • A.4 Signed HealthBench criteria during training and evaluation: HealthBench criteria use signed integer points, with positive weights rewarding desired behavior and negative weights penalizing undesired behavior such as unsafe advice.POW3R’s category-normalized aggregation assumes non-negative weights because its normalizers and dynamic-factor renormalization treat weights as non-negative priors.
  • A.4 Signed HealthBench criteria during training and evaluation: For training, negative HealthBench criteria are rewritten as positive avoidance criteria so category normalization and dynamic factors remain well-defined.The judge prompt and category labels are adjusted to score avoidance rather than occurrence.
  • A.4 Signed HealthBench criteria during training and evaluation: Evaluation retains HealthBench’s original scoring script, including the HB Strict column, HB overall score, and every other reported HB number.The training-time rewrite is not exposed to evaluation.

B Judge selection · C System prompts

Judge selection is treated as a first-class experimental decision: candidate per-rubric judges are calibrated against a GPT-5.4 high-effort reference, then assigned to training or held-out evaluation using an explicit cost–quality trade-off. The accompanying system prompts enforce independent, criterion-level JSON verdicts and document calibration-only alternatives.

  • B Judge selection: Judge choice directly affects every downstream comparison, so the paper calibrates candidate judges against a high-effort reference and selects operating points by cost–quality trade-off.The reference judge is GPT-5.4 with high-effort reasoning and per-criterion explanations enabled, although the paper does not treat one judge as absolute ground truth.
  • B Judge selection: 1,000 stratified task–rollout–rubric triples from MM-train are scored by the reference and every candidate across six rubric categories and weight bins.All candidates score the same items, enabling paired, apples-to-apples agreement comparisons across easy- and hard-to-judge regions.
  • B Judge selection: 95.4% sets the reference judge’s agreement ceiling against a human-rated calibration subset, while candidate rows measure agreement with the reference on all 1,000 calibration items.Table 5 marks the operating points used in the paper in bold.
  • B Judge selection: Removing explanations can preserve aggregate agreement but disrupt per-model rank correlation, while per-category judging is cheaper but merges or skips independent rubric verdicts.The paper therefore retains explanations and uses per-rubric judging throughout; mini-class judges are roughly 12× more expensive than nano-class judges per 1,000 calls.
  • B Judge selection: 91.4% agreement at $0.12 / 1,000 judgments defines the GPT-5.4-nano medium-reasoning-with-explanation training judge, while 93.6% agreement at $1.52 / 1,000 judgments defines the GPT-5.4-mini evaluation judge.Using the more accurate judge only for evaluation reduces judge–training entanglement.
  • C System prompts: Both operating-point judges use the same single-criterion template, differ only in the underlying model, run at temperature T=1.0, and allow up to 2048 completion tokens.Training uses GPT-5.4-nano with medium reasoning; held-out evaluation uses GPT-5.4-mini with medium reasoning.
  • C System prompts: Each per-criterion prompt supplies rubric metadata and the response, instructs the judge to evaluate only that criterion, and returns one-sentence reasoning followed by a binary JSON verdict.The rules distinguish explicit versus implicit, objective versus subjective, and exact OCR/text recognition criteria; rubric weight is context only and does not affect pass/fail.
  • C System prompts: The appendix also specifies verdict-only and per-category batched prompts as calibration-only variants, while the paper judges every rubric criterion one at a time.All three prompt forms are reproduced verbatim from the training and evaluation code, apart from presentation formatting.

D Training hyperparameters · E Rubric categories with example criteria

Training uses shared hyperparameters across all reported policies and reward constructions, while MM rubrics organize response quality into six balanced categories with explicit operational criteria. The categories span visual grounding, reasoning, completeness, instruction following, truthfulness, and presentation.

  • D Training hyperparameters: All reported GRPO and POW3R runs use the same hyperparameters across base policies and reward constructions.This applies to Qwen3-VL-4B, Qwen3-VL-8B, Gemma3-4B, Qwen3-4B, and Qwen3-8B across MM and HB settings.
  • E Rubric categories with example criteria: MM rubrics define six high-level quality categories for balancing grounding, reasoning, completeness, instruction following, factual correctness, and presentation.The categories are Visual perception, Visual reasoning, Content completeness, Instruction following, Truthfulness, and Writing style / presentation.
  • E Rubric categories with example criteria: Each MM task typically contains 7–10 criteria distributed across the six categories, with tags encoding weight, explicitness, and objectivity.The schema uses the form [weight | explicit/implicit | objective/subjective].
  • E Rubric categories with example criteria: Visual perception criteria assess image-readable content such as objects, text, named entities, and counts.Examples require identifying visible basketball players and listing stores named in a letter.
  • E Rubric categories with example criteria: Visual reasoning criteria require combining visual cues or contextual evidence to infer versions, structures, or authorship and requests.Examples include identifying NBA 2K20 from screenshot details and inferring a letter’s author and request from its postscript.
  • E Rubric categories with example criteria: Content completeness and instruction following evaluate whether required elements are covered and constraints on order, sectioning, length, format, or procedure are respected.Examples require listing all labeled objects, explaining effects on reactive and active power, preserving a specified order, and answering in 2 sections.
  • E Rubric categories with example criteria: Truthfulness and writing style / presentation assess factual accuracy, unsupported claims, organization, formatting, and audience-appropriate language.Examples include correctly identifying Avatar: The Last Airbender, explaining impedance effects, alphabetizing a bullet list, and using simple high-school-level language.

F Qualitative examples

Across four held-out multimodal tasks, POW3R’s dynamic reward produces more rubric-compliant outputs than the base policy and static reward settings. The examples span spatial reasoning, visual perception, polar-curve identification, and visual-grounded worksheet generation.

  • Example 2: Visual perception: 5/5 passed: POW3R identifies the meme animal as a dog and connects its indifferent expression to the Google Home obedience-but-not-understanding joke.Base, Rscalar, and Rcat misidentify the dog as a pig and pass 0/5 required rubrics.
  • Example 3: Polar-curve identification: 5/8 passed with strict = True: POW3R names the second polar-curve shape as a cardioid and identifies its single rightward cusp and heart-shaped contour.Base, Rscalar, and Rcat pass 0–3/8 rubrics and strict = False while hedging among several shape names.
  • Example 4: Visual-grounded worksheet generation: 8/13 passed with strict = True: POW3R generates four fruit-based arithmetic problems, each evaluating to 4, while mixing division, multiplication, and subtraction.Base, Rscalar, and Rcat pass 4–5/13 rubrics and strict = False, with at least one incorrect answer and mid-stream self-correction.
Loading 2605.20164v1…