PaperlayerView exact revision on arXiv

Source-linked AI summary

Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling

Lipeng Xie, Sen Huang, Zhuo Zhang, Anni Zou, Yunpeng Zhai, Dingchao Ren, Kezun Zhang, Haoyuan Hu, Boyin Liu, Haoran Chen, Zhaoyang Liu, Bolin Ding

arXiv:2510.17314v2cs.LGcs.AI

TL;DR

Conventional reward modeling places human preferences in opaque neural weights and often requires large datasets. This paper searches natural-language rubrics through training-free iterative induction and compression. With 70 preference pairs, its rubric-guided judges outperform a specialized fully trained reward model on reported benchmarks.

  • Problem

    Conventional reward models encode preferences in opaque neural weights and require massive datasets to perform reliably, while their rationale can be difficult to diagnose.

    Source: §1 · p. 1

  • Method

    The framework iteratively induces criteria through verification-driven refinement, compresses them with an information-theoretic coding-rate objective, and organizes the core set as hierarchical rubrics.

    Source: §1 · p. 2

  • Results

    80.91% on RewardBench2 and 88.28% on RM-Bench for rubric-guided Qwen3-8B, versus 78.20% and 82.60% for Skywork-Reward-V2-Qwen3-8B, respectively.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The reported results indicate that explicit rubric parameterization can offer a data-efficient alternative to implicit neural learning without extensive parameter updates.

    Source: §4.2 · p. 7

    The framework's absolute performance remains bounded by backbone capabilities, general-chat rubrics may miss specialized-domain nuances, and test-time scaling adds inference cost.

    Source: A Limitations · p. 13

Abstract

from arXiv · show

Conventional reward modeling relies on gradient descent over neural weights, creating opaque, data-hungry "black boxes." We propose a paradigm shift from implicit to explicit reward parameterization, recasting optimization from continuous weight spaces to the discrete space of natural language rubrics. We introduce a training-free framework based on iterative rubric learning: it locally induces discriminative criteria via verification-driven refinement, and globally compresses the candidate criteria pool into a compact core set by maximizing an information-theoretic coding rate objective. We organize the compressed core set into a hierarchical rubric structure -- high-level evaluation dimensions supported by concrete verification checks -- serving as an interpretable, portable reward function. Empirically, our approach challenges prevailing data scaling assumptions: using only 70 preference pairs, our rubric-guided judges outperform fully trained reward models on diverse benchmarks. For instance, Qwen3-8B equipped with our learned rubrics achieves 80.91% on RewardBench2, surpassing the specialized Skywork-Reward-V2-Qwen3-8B (78.20%). These results demonstrate that alignment signals are highly compressible and can be effectively captured through explicit symbolic search.

1 Introduction

The introduction frames opaque, data-hungry reward models as a motivation for explicit, interpretable rubric-based reward parameterization. It presents training-free iterative rubric learning and reports strong performance with only 70 preference pairs.

1 Introduction

  • Conventional reward models encode human preferences in opaque neural weights, require massive datasets to perform reliably, and can be susceptible to reward hacking.Their opacity also makes the rationale behind model preferences difficult to diagnose.Source: §1 · p. 1
  • The paper reframes reward modeling as discrete search over human-readable natural-language rubrics rather than gradient descent over neural weights.The rubrics are intended as a transparent, auditable alternative to black-box reward models.Source: §1 · pp. 1–2
  • Local induction treats single-sample learning as active hypothesis testing, using verification-driven refinement to produce criteria that explain observed preferences.The method targets validity and discriminative power at the instance level.Source: §1 · p. 2
  • Global compression embeds induced rubrics and maximizes coding rate to penalize redundancy while promoting diversity in a compact core set.The paper describes compressing thousands of local rubrics into a minimal, high-coverage set.Source: §1 · p. 2
  • The resulting rubric set organizes high-level evaluation dimensions with concrete verification criteria as a portable, interpretable reward model that requires no gradient updates.The iterative algorithm alternates local induction and information-theoretic compression to discover criteria from minimal data.Source: §1 · p. 2
  • 80.91% on RewardBench2 with 70 preference pairs, versus 78.20% for Skywork-Reward-V2-Qwen3-8B trained on 26 million pairs.The introduction describes this as an approximately 370,000× reduction in data requirements.Source: §1 · p. 2

2 Related Work

The related work contrasts latent or manually supplied criteria with approaches that infer explicit rubrics from data. It positions this paper's iterative verification and compression as a training-free route to discriminative, non-redundant rubrics.

LLM-as-a-Judge with Latent Criteria

  • LLM-as-a-Judge approaches often leave criteria latent in model weights, while interpretable alternatives commonly require extensive training or do not separate criteria from reasoning.Unstructured prompts can yield inconsistent judgments; cited alternatives include fine-grained feedback and Chain-of-Rubrics.Source: §2 · p. 3

Explicit Reward Parameterization

  • Training-based explicit-reward methods incur high training costs and may assume unverified external rubrics, while expert-curated rubric methods scale poorly.The paper positions its approach as inferring explicit parameters from data without gradient updates.Source: §2 · p. 3

Automated Rubric Discovery

  • Compared with OpenRubrics' 35.7k synthetic pairs, this framework uses 70 preference pairs and alternates verified induction with information-theoretic compression.The related-work passage describes this as a 500× reduction and says the learned rubrics are discriminatively valid and non-redundant.Source: §2 · p. 3

3 Methodology

The methodology replaces continuous reward-weight optimization with discrete natural-language rubric selection, developed through iterative induction and compression and assessed with rubric-level quality metrics.

3.1 Formulation

  • The preference model is parameterized by a natural-language rubric set, but its discrete, combinatorially large search space motivates accuracy-based and local-then-global relaxations.The method replaces log-likelihood with a 0-1 accuracy surrogate and induces candidates locally before selecting a compact core set globally.Source: §3.1 · pp. 3–4

3.2 Iterative Rubric Learning

  • Local Induction uses iterative hypothesis refinement to find rubrics that explain why a preferred response is better, verifying candidates and revising failures.Refinement continues until verification succeeds or reaches a maximum number of steps; induced rubrics enter the candidate pool.Source: §3.2 · p. 5
  • Global Compression selects a compact subset from the noisy, overlapping candidate pool by maximizing coding rate, which favors semantic diversity over cluster compactness.A greedy selection stops when marginal gain falls below a threshold; normalization penalizes redundant additions and permits information-saturation detection.Source: §3.2 · p. 5
  • The iterative workflow alternates induction and compression until the core set stabilizes, then organizes criteria into evaluation dimensions supported by concrete verification checks.This hierarchy is intended to make abstract assessments consistently evidence-based.Source: §3.2 · pp. 5–6

3.3 A Framework for Rubric Analysis

  • Coverage measures how often a rubric gives a discriminative signal, precision measures its conditional accuracy, and contribution measures the accuracy drop when it is removed.The analysis framework reports that the learned set balances general, high-coverage rubrics with specialized, high-precision ones.Source: §3.3 · p. 6

4 Experiments

Across four benchmarks, the experiments evaluate rubric-guided judges, data efficiency, component ablations, and downstream policy optimization. The results report strong comparisons with trained reward models, rapid information saturation, and gains from verification, compression, and hierarchical structure.

4.1 Experimental Setup

  • The setup uses HelpSteer3-Preference and UltraFeedback-Binarized rubrics and evaluates on RewardBench, RewardBench2, RM-Bench, and JudgeBench across chat, reasoning, coding, and safety.The benchmark suite includes zero-shot and prompting baselines, expert-curated judges, and fully trained reward models.Source: §4.1 · pp. 6–7, §4 · p. 6

4.2 Main Results

  • Rubric guidance raises RewardBench2 scores by +6.54% for Qwen3-8B and +6.72% for Qwen3-32B; Claude-4-Sonnet reaches 95.81% on RewardBench and 86.90% on RewardBench2.The RewardBench result is +1.20% for Claude-4-Sonnet; similar gains are reported for Llama-3.1-8B.Source: §4.2 · p. 7
  • The training-free rubric-guided Qwen3-8B judge achieved 80.91% accuracy on RewardBench2 and 88.28% on RM-Bench, exceeding Skywork-Reward-V2-Qwen3-8B’s respective 78.20% and 82.60%.Source: §4.2 · p. 7
  • Across HelpSteer3 and UltraFeedback, average performance differs by less than 1.2% for each reported judge, while RM-Bench gains span Chat (+6.63%), Math (+2.60%), Code (+2.05%), and Safety-Refuse (+1.96%).The overall RM-Bench improvement is +2.44%; Safety-Response shows marginal changes.Source: §4.2 · p. 7

4.3 Data Efficiency and Convergence Analysis

  • Locally induced rubrics exceed 90% accuracy after 3–5 refinement iterations, with each preference pair supplying a high-quality signal.The authors attribute the per-example discrimination to verification-driven induction.Source: §4.3 · pp. 7–8
  • Figure 3 shows early selections sparsely covering the semantic manifold, while marginal coding-rate gain turns negative at Batch 6 and the process stops at Batch 7 (70 pairs).The authors interpret this as information saturation: redundancy penalty outweighs expansion, and remaining data largely repeats evaluation principles.Source: §4.3 · p. 8, Figure 3 · p. 9

4.4 Ablation Studies

  • Full iterative refinement beats single-pass generation by +2.43% on RewardBench2 and +2.04% on RM-Bench; coding-rate maximization beats random selection by +3.16% on RewardBench2, and hierarchy beats a flat list by +1.13%.Blind revision without verification gains +2.14% on RewardBench2 but falls by -0.28% on RM-Bench; the ablations use Qwen3-32B with HelpSteer3 rubrics.Source: §4.4 · pp. 8–9, Table 2 · p. 9

4.5 Application to Policy Optimization

  • After style control, rubric-guided policy optimization achieves a +2.8% gain and a 57.0% Arena-Hard win-rate, while Skywork drops by -4.8%.On AlpacaEval, the method reports a 42.2% length-controlled win-rate and 42.8% vanilla performance.Source: §4.5 · pp. 9–10, Table 3 · p. 10

5 Conclusion

The conclusion presents explicit rubric learning as an alternative to implicit neural reward modeling. It reports interpretable hierarchical rubrics distilled from as few as 70 samples, with performance rivaling or exceeding fully trained reward models.

5 Conclusion

  • The framework replaces implicit neural weight optimization with discrete search over rubrics, using local induction and information-theoretic compression without gradient updates.The resulting hierarchical rubrics make evaluation criteria human-readable and verifiable.Source: §5 · p. 10

A Limitations

The framework’s limitations concern backbone capability, domain specialization, and inference cost; its hyperparameters govern sample-level refinement and rubric-set compression.

A Limitations

  • The framework’s performance remains bounded by backbone reasoning and instruction-following, general-chat rubrics may miss specialized domains, and test-time scaling increases inference cost.The authors identify domain-targeted induction as a need for specialized settings and describe a precision–cost trade-off for Voting@5.Source: A Limitations · p. 13

B Algorithm Hyperparameters

  • Emax counts self-correction iterations for one sample, not passes over the dataset.Convergence typically occurs within 3–5 iterations.Source: B Algorithm Hyperparameters · p. 13
  • The ε parameter controls negative information gain; its penalty eventually outweighs semantic expansion as the core rubric set grows.This triggers early termination of compression.Source: B Algorithm Hyperparameters · p. 13

C Extended Experimental Details

The experiments use Qwen3-32B to generate rubrics, then evaluate them across multiple judges and benchmarks under fixed rubric and voting protocols. Qwen3-32B rubrics show the strongest cross-model compatibility on RewardBench2.

C Extended Experimental Details

  • Rubrics are generated with Qwen3-32B and evaluated using Qwen3 open-weights models from 8B to 235B, Claude-4-Sonnet, and GPT-4o.A sensitivity analysis comparing Qwen and Llama generation backbones is provided in Appendix I.Source: C Extended Experimental Details · p. 13
  • Each source dataset uses one fixed rubric set across benchmarks without task-specific tuning, and prompt-based methods use identical judge models.Voting is standardized at RewardBench2 Voting@10, RewardBench and JudgeBench Voting@5, and RM-Bench Voting@1.Source: C Extended Experimental Details · p. 13
  • Open-weights inference uses temperature 0.7, top p 0.8, and top k 20, with think mode enabled for induction and evaluation.Proprietary models use standard API endpoints with default parameters.Source: C Extended Experimental Details · p. 14
  • Rubric learning processes 70 pairs with up to 10 refinement iterations, with one-time overhead described as negligible beside reward-model training.Evaluation cost is dominated by voting, and all methods use the same voting strategy.Source: C Extended Experimental Details · p. 14
  • Policy optimization fine-tunes Qwen2.5-7B-Instruct on WildChat prompts labeled by the rubric-guided HelpSteer3-source judge.Training uses 2 epochs, global batch size 1024, sequence length 2048, learning rate 5e-7, and 2 × 8 H20 nodes.Source: C Extended Experimental Details · p. 14

D Analysis on Cross-Model Compatibility of Rubrics

  • Qwen3-32B-generated rubrics show the strongest cross-model compatibility on RewardBench2, including a +7.0% gain for GPT-4o, while Claude-4-Sonnet has the highest evaluator scores.Rubrics improve over the no-rubric baseline across all evaluators; Claude-4-Sonnet-generated rubrics are less effective with other models.Source: D Analysis on Cross-Model… · p. 14, Figure 4 · p. 14

E Local Induction Convergence and Test-Time Scaling Analysis

Verification-driven refinement converges quickly, while rubric guidance retains an advantage across voting budgets and is especially effective on ambiguous ties. The reported cross-domain analysis finds gains in most RM-Bench domains, with a slight decline on Safety-Response.

E Local Induction Convergence and Test-Time Scaling Analysis

  • Rubric guidance maintains a 6–7 point RewardBench2 advantage across voting strategies, with an approximately 25-point improvement on ambiguous ties.The authors characterize rubric benefits as additive to ensemble voting.Source: Figure 5 · p. 15, E Local Induction Convergence and… · p. 15
  • Local induction reaches 99.20% accuracy on UltraFeedback and 95.80% on HelpSteer3, with both curves plateauing around iteration 6.The first 2–3 iterations bring steep gains, and subsequent gains after the plateau are under 1%.Source: E Local Induction Convergence and… · p. 15, Figure 5 · p. 15
  • Voting yields the largest gains from Voting@1 to Voting@5, with diminishing returns afterward; Voting@5 to Voting@10 is described as a practical accuracy–cost trade-off.The largest improvement from rubric guidance is on the ties subset, where the baseline struggles to make decisive judgments.Source: E Local Induction Convergence and… · p. 15

F.1 Cross-Domain Applicability

  • On RM-Bench, general conversational rubrics improve Chat by 6.63%, Math by 2.60%, Code by 2.05%, and Safety-Refuse by 1.96%, while Safety-Response declines by 0.07%.The authors suggest nuanced safety scenarios may require domain-specific criteria.Source: §F.1 · p. 15

F.2 Difficulty-Stratified Analysis (RM-Bench)

Rubric guidance produces larger gains on difficult RM-Bench samples and improves RewardBench2 evaluation dimensions, especially ties. The core rubric analysis finds both broad-coverage and specialized criteria contribute measurable value.

F.2 Difficulty-Stratified Analysis (RM-Bench)

  • Hard RM-Bench samples improve by 4.68% versus 2.44% overall, with the largest hard-sample domain gain in Chat at 13.95%.Hard-sample gains also occur in Math (+4.54%) and Safety-Refuse (+3.64%).Source: §F.2 · pp. 15–16, Table 5 · p. 16

F.3 Evaluation Dimension Analysis (RewardBench2)

  • On RewardBench2, rubric guidance improves overall performance by 6.72%, from 75.55% to 82.27%, and raises Ties performance by 25.49%, from 56.86% to 82.35%.The Safety domain improves by 10.34%.Source: Table 6 · p. 16, §F.3 · p. 16

G Analysis of Core Rubrics

G.1 Quantitative Analysis

  • “Clarity and Structured Presentation” has 97.92% coverage and a 7.09% removal-related accuracy drop, while “Narrative and Context Fidelity” has 71.91% coverage and 68.24% precision.Positive contribution scores across all dimensions indicate each selected dimension adds measurable value.Source: §G.1 · p. 16

G.2 Qualitative Analysis: Semantic Mapping

Rubric-guided judges map abstract evaluation dimensions to domain-specific verification logic rather than matching keywords mechanically. Examples show this mapping catching implementation-interface errors, scrutinizing mathematical process, and preserving culturally specific conversational details.

G.2 Qualitative Analysis: Semantic Mapping

  • Abstract dimensions map to domain-specific verification logic through semantic analogy, not mechanical keyword matching.The analysis examines examples spanning chat, code, and math.Source: §G.2 · p. 16
  • For code, Strict Compliance with Requirements led the judge to flag a C++ return-type mismatch that the baseline missed.The implementation returned vector<float> instead of the requested pair, violating the interface contract.Source: §G.2 · p. 17
  • For math, Factual Accuracy and Context Consistency led the judge to penalize premature rounding despite an approximately correct final answer.The rubric emphasized process fidelity and precision in intermediate steps.Source: §G.2 · p. 17
  • For a recipe request, Narrative and Context Fidelity prompted attention to culturally authentic detail, such as “sweet soy sauce” for Indonesian cuisine.The example illustrates how a general rubric can capture domain-specific nuance.Source: §G.2 · p. 17

H Multi-Seed Stability Analysis

Across five runs, rubric-guided evaluation shows low variance, while performance gains exceed that variance on both RewardBench2 and RM-Bench. The baseline evaluation harness is itself stable, and the rubric sets vary somewhat more because induction samples different data.

H Multi-Seed Stability Analysis

  • Base models show SD ≈0.1%, indicating a stable evaluation harness with minimal inference-time stochasticity.The pipeline was run five times with different seeds and distinct HelpSteer3 preference-data subsets.Source: H Multi-Seed Stability Analysis · p. 17, Table 8 · p. 17
  • Rubric-guided results have SD ≈0.3%–0.4%, with standard deviations below 0.45 across seeds.The authors attribute the slightly higher variance to data sampling during induction and report consistent convergence to high-quality rubrics.Source: H Multi-Seed Stability Analysis · p. 17, Table 8 · p. 17
  • On RewardBench2 (Qwen3-32B), the +6.53% gain is approximately 19× the 0.35% standard deviation; on RM-Bench, +2.34% is over 10× the 0.22% standard deviation.Across all runs, the minimum “Ours” score exceeds the maximum “Base” score.Source: H Multi-Seed Stability Analysis · p. 17, Table 8 · p. 17

I Backbone Sensitivity: Qwen vs. Llama

Rubrics improve both Qwen3-8B and Llama-3.1-8B across benchmarks, but absolute performance remains bounded by backbone capability. The appendix also presents induced rubric collections from two datasets and prompts for induction, evaluation, revision, and hierarchical organization.

I Backbone Sensitivity: Qwen vs. Llama

  • Llama-3.1-8B gains +3.59% on RewardBench versus +0.57% for Qwen3-8B, while RewardBench2 scores remain 46.23% versus 80.91%, respectively.The authors characterize rubrics as a performance multiplier rather than a substitute for base-model capability.Source: I Backbone Sensitivity: Qwen vs. Llama · p. 18, Table 9 · p. 18

K Induced Rubric Collections

K.1 HelpSteer3-Preference Dataset

  • HelpSteer3-Preference rubrics include dimensions for factual accuracy, requirement compliance, clarity, comprehensive coverage, and narrative/context fidelity.The listed checks address factual verification, explicit constraints, structure, completeness, and continuity.Source: §K.1 · pp. 18, 20

K.2 UltraFeedback-Binarized Dataset

  • UltraFeedback-Binarized rubrics cover factual/domain grounding, requirement fulfillment, logical organization, contextual depth, and ethical responsibility/user alignment.Their checks include source restrictions, valid formats, actionable detail, and transparent uncertainty.Source: §K.2 · pp. 20–22

L Prompt Templates

  • The prompt templates cover local rubric induction from one preference pair, rubric-guided pairwise evaluation, verification-feedback revision, and hierarchical organization.The four templates are shown in Figures 6–9.Source: Figure 6 · p. 22, Figure 7 · p. 23, Figure 8 · p. 24

Related papers

Loading 2510.17314v2…