PaperlayerView exact revision on arXiv

Source-linked AI summary

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

Licheng Pan, Haocheng Yang, Haoxuan Li, Yuzhou Chen, Yichen Sun, Yuan Lu, Xiaoxi Li, Hao Wang, Zhixuan Chu, Kui Ren

arXiv:2610.21214v1cs.AI

TL;DR

Rubric-based LLM evaluation typically gives equal weight to binary judgments even though judgment certainty varies by criterion, leaving the aggregation without a per-criterion uncertainty signal. URW uses paraphrased judgment templates and split-conformal calibration to produce criterion weights, and experiments report improved preference accuracy and downstream RL alignment. The guarantee assumes exchangeability, and the method captures only uncertainty revealed by its fixed template pool.

  • Problem

    Uniform rubric aggregation treats binary criterion judgments as equally reliable despite substantial variation in judgment difficulty.

    Source: §1 · p. 1, §2.1 · p. 2

  • Method

    URW averages deterministic judgments across paraphrased templates and converts the resulting criterion-level signal into split-conformal p-value weights.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 4

  • Results

    URW consistently improves pairwise preference accuracy and downstream RL alignment across diverse preference benchmarks and verifier families.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors report URW as a training-free, distribution-free way to weight criteria while controlling the chance of spuriously downweighting correct judgments.

    Source: §1 · p. 2

    URW’s guarantee relies on calibration–test exchangeability, which can be strained by RLVR distribution shifts, and its fixed paraphrase pool does not capture other uncertainty sources such as rubric or task ambiguity.

    Source: §5 · p. 9

Abstract

from arXiv · show

Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a binary judgment from an LLM verifier. However, judgment certainty varies substantially across criteria. Existing methods uniformly aggregate these judgments into a final score, implicitly treating every criterion as equally certain and overlooking this heterogeneity. As a result, uncertain judgments remain incorporated into reward estimation without distinction, potentially yielding misleading training signals for downstream reinforcement learning (RL). To address this limitation, we propose Uncertainty-aware Rubric Weighting (URW), a training-free framework that quantifies criterion-level judgment uncertainty via conformal calibration. Specifically, URW elicits a soft uncertainty signal from a fixed pool of paraphrased judgment templates and converts it into a per-criterion weight using split-conformal $p$-values with provable coverage guarantees. Extensive experiments demonstrate that URW consistently improves pairwise preference accuracy and delivers stronger downstream RL alignment.

1 Introduction

Rubric-based LLM evaluation commonly aggregates binary criterion judgments uniformly, despite substantial differences in how reliably criteria can be judged. URW adds training-free, conformal-calibrated criterion weights, and experiments report improved preference accuracy and downstream RL alignment.

Problem

  • Uniform aggregation gives equally weighted binary judgments despite criteria varying in difficulty, allowing one uncertain subjective judgment to invert a close preference.Format checks are typically clear-cut, while tone, helpfulness, and content-quality judgments are often subjective.Source: §1 · p. 1

Research gap

  • Prior uncertainty methods do not provide training-free, coverage-controlled per-criterion weights compatible with deterministic, single-verifier rubric aggregation.Existing approaches instead use response-level uncertainty, sampling or verifier committees, or trained modules requiring annotated data.Source: §1 · pp. 1–2

Contribution

  • URW uses paraphrased judgment templates and split-conformal p-values to create training-free, distribution-free per-criterion weights with a guarantee against spuriously downweighting correct judgments.The paper reports consistent gains in pairwise preference accuracy and downstream RL alignment across five preference benchmarks.Source: §1 · p. 2

Evaluation

  • Experiments on five preference benchmarks show consistent pairwise-accuracy improvements over diverse baselines and stronger downstream RL alignment.The paper presents these evaluations as testing URW’s efficacy.Source: §1 · p. 2

2 Preliminaries

Rubric-based preference scores aggregate binary judgments across prompt-specific criteria, while split conformal prediction provides a held-out, exchangeability-based approach to uncertainty calibration. The paper frames its task as replacing uniform criterion weights with instance-adaptive weights that reflect judgment certainty.

2.1 Rubric-Based Preference Evaluation

  • Rubric evaluation assigns binary judgments to prompt-specific criteria and combines them through a weighted linear score, while uniform weights treat all judgments as equally reliable.Subjective criteria such as tone or helpfulness can be harder to judge than objective format or length checks.Source: §2.1 · p. 2

2.2 Split Conformal Prediction

  • Figure 1 illustrates how a single uncertain judgment on a subjective criterion can invert the aggregate preference under uniform weighting.The example contrasts responses that differ in tone while otherwise tying or splitting across criteria.Source: Figure 1 · p. 3
  • Split conformal prediction uses a held-out calibration set and, under exchangeability, provides finite-sample marginal coverage without retraining the black-box predictor.The section defines nonconformity from the verifier–gold-label discrepancy and uses a miscoverage rate to form prediction sets.Source: §2.2 · p. 3

2.3 Problem Definition

  • The problem is to construct instance-adaptive criterion weights from a deterministic verifier’s binary outputs, which expose no direct confidence signal.The weights should let more uncertain judgments exert less influence in response-level aggregation.Source: §2.3 · p. 3

3 Methodology

URW probes a deterministic verifier with meaning-preserving paraphrases to estimate criterion-level uncertainty, then calibrates that signal into conformal weights. The resulting weights discount judgments near the decision boundary, with a validity guarantee under exchangeability.

3.1 Motivation

  • With uniform weights, an uncertain judgment on one subjective criterion can tip a close preference; higher-temperature sampling is unsuitable for deterministic verification and stable reward signals.Prior approaches also use response-level uncertainty, verifier committees, or trained modules rather than training-free criterion-level weights.Source: §3.1 · pp. 3–4

3.2 Quantifying Verifier Uncertainty via Template Ensembling

  • URW applies deterministic decoding to semantics-preserving paraphrased judgment templates and averages their binary votes into a soft criterion score.The score approaches 0 or 1 for template agreement and 0.5 for maximal uncertainty.Source: §3.2 · p. 4

3.3 Calibrating Verifier Uncertainty via p-value Weighting

  • Split-conformal p-values of the voted labels become raw criterion weights, which increase with template agreement and are normalized for weighted aggregation.Under exchangeability of calibration and test triples, the guarantee controls spuriously low weights on correct votes uniformly over α, without requiring α to be fixed in advance.Source: §3.3 · pp. 4–5

4 Experiments

The experiments assess URW against weighting baselines across preference aggregation, ablations, generalization, template-count sensitivity, and downstream RLVR.

4.1 Experimental Setup

  • Evaluation uses five preference benchmarks and three frozen verifiers; calibration uses about 2k rubric annotations from 200 domain-stratified prompts, disjoint from evaluation splits.RubricBench and ResearchQA use human-sourced rubrics, while the other benchmarks use generated rubrics.Source: §4.1 · p. 5

4.2 Overall Performance (RQ1)

  • URW outperforms all baselines across five benchmarks and three verifiers in pairwise preference accuracy.The results are attributed to calibrated p-value weighting that retains continuous uncertainty information rather than using binary hard abstention.Source: Table 1 · p. 6, §4.2 · p. 6

4.3 Ablation Study (RQ2)

  • URW† shows template ensembling improves aggregation, URW‡ shows conformal calibration improves weighting, and the complete URW outperforms both ablations.URW† improves over Uniform by up to 3.9% on ResearchQA; URW‡ improves by 2.1%–9.2% across benchmarks.Source: §4.3 · pp. 6–7, Table 2 · p. 7

4.4 Generalization Studies (RQ3)

  • Across Qwen 3 and GLM 4 verifier families, covering 8B–32B models, URW outperforms Uniform on all five benchmarks and all 20 verifier–benchmark comparisons, with gains of 0.6–15.1 relative accuracy points.URW integrates with diverse verifier families without family-specific tuning.Source: §4.4 · p. 7
  • On RubricBench and ResearchQA, URW is best in every human-audited and LLM-generated rubric configuration, improving over Uniform by 1.6%–13.6%.LLM-generated rubrics lower absolute accuracy for all methods and benchmarks, but URW remains best without source-specific retuning.Source: §4.4 · pp. 7–8, Table 3 · p. 8, Table 4 · p. 8

4.5 Hyperparameter Sensitivity Study (RQ4)

  • Across displayed datasets and verifiers, multiple templates beat N = 1 Uniform; most gains arrive with a small pool, with later changes smaller and sometimes within 15% confidence intervals.The displayed results cover RubricBench and ResearchQA; results on three other datasets are in Appendix D.1.Source: Figure 3 · p. 8, §4.5 · p. 8

4.6 Downstream RLVR Performance (RQ5)

  • URW leads all four policy–benchmark settings; for GLM 4 9B, it reaches 48.7 Arena-Hard win rate and 50.0 GPQA-Diamond accuracy, versus 42.4 and 43.4 under Mondrian-SCP.Policies were trained for an identical budget of 150 steps and evaluated zero-shot.Source: §4.6 · pp. 8–9, Table 6 · p. 9, Table 5 · p. 9

5 Conclusion

URW addresses heterogeneous uncertainty in rubric judgments with a training-free, distribution-free framework that uses paraphrased templates and split-conformal p-values to create continuous criterion-level weights. The paper also situates URW as a way to calibrate the reliability of individual judgments within existing rubric and verifier pipelines.

5 Conclusion

  • URW ensembles paraphrased judgment templates and calibrates their agreement with split-conformal p-values to construct continuous criterion-level weights.Its theoretical guarantee controls the probability of assigning a spuriously low weight to a correct judgment under exchangeability.Source: §5 · p. 9
  • URW addresses uncertainty revealed by a fixed pool of semantics-preserving templates, but does not separately identify rubric or task ambiguity.Its theoretical guarantee relies on exchangeability between calibration and test triples, which can be strained by distribution shifts during RLVR training.Source: §5 · p. 9

A Related Work

  • URW retains the rubric and verifier pipeline while calibrating each criterion judgment's reliability before aggregation.The related-work framing describes URW as converting cross-template stability of a frozen binary verifier into a split-conformal criterion-level weight with a distribution-free validity guarantee.Source: A Related Work · p. 13

A.1 Rubric-Based LLM Evaluation

Rubric-based evaluation decomposes open-ended output quality into explicit criteria that are verified separately, and rubrics increasingly support post-training and agent improvement. URW complements this pipeline by calibrating criterion-level reliability before aggregation, using a frozen deterministic verifier and held-out labeled data.

A.1 Rubric-Based LLM Evaluation

  • Rubric-based evaluation decomposes open-ended quality into explicit, verifiable criteria, offering an alternative to holistic LLM judging.Rubrics are used in evaluation and generation, and benchmarks develop instance-specific fine-grained criteria.Source: §A.1 · p. 13
  • Rubric construction has advanced through self-evolving generation and structured meta-rubrics that compile requirements into machine-gradable binary checks.The passage cites Chen et al. (2026b) for self-evolving generation and Chen et al. (2026a) for structured meta-rubrics.Source: §A.1 · p. 13
  • Rubric judgments provide rewards for post-training and are increasingly used for agent improvement in domains without exact verifiers.The passage cites Mu et al. (2024), Gunjal et al. (2026), and Shan & Shao (2026) for post-training rewards, and Wang et al. (2026c) and Liu et al. (2026) for agent improvement.Source: §A.1 · p. 13
  • Rubric-based methods commonly aggregate criterion judgments with fixed weights, while URW calibrates each criterion's reliability before aggregation.URW retains the underlying rubric and verifier pipeline.Source: §A.1 · p. 13

A.2 Uncertainty Estimation for LLM Judges

  • Conformal methods for LLM evaluation have produced response-level intervals, sets, or abstention decisions, whereas URW uses each voted rubric label's conformal p-value as a continuous aggregation weight.Split conformal prediction provides finite-sample marginal validity under exchangeability.Source: §A.2 · p. 13
  • Decoding-time consistency and heterogeneous judge committees estimate confidence but require nonzero-temperature sampling or multiple verifier models.URW instead uses a single frozen deterministic verifier and a fixed pool of semantic-preserving templates.Source: §A.2 · p. 13
  • Dedicated uncertainty-aware evaluators and calibration probes require training, whereas URW calibrates its criterion-level signal on held-out labeled data without training an additional evaluator.This design makes the uncertainty estimate directly compatible with weighted rubric aggregation.Source: §A.2 · p. 13

B Theoretical Justification

The theoretical argument establishes a finite-sample validity guarantee for URW weights under exchangeability with a fixed verifier and template pool. The guarantee concerns correct voted judgments receiving spuriously low weights; it does not ensure incorrect judgments receive low weights.

Theorem 3.1

  • Under exchangeability of calibration and test triples with the same fixed verifier and template pool, Theorem 3.1 gives finite-sample validity for the URW uncertainty weight.The proof applies the split-conformal rank argument to exchangeable nonconformity scores.Source: B Theoretical Justification · pp. 13–14
  • The proof defines nonconformity as the absolute difference between the template-averaged score and the true label, then ranks the test score against calibration scores.The split-conformal p-value uses the count of calibration scores at least as large as the test score, with a +1 numerator and n + 1 denominator.Source: B Theoretical Justification · pp. 13–14
  • The guarantee only bounds how often a correct voted judgment receives a spuriously low weight; it does not guarantee low weights for incorrect judgments.Whether incorrect or uncertain judgments receive low weights depends on the empirical separation of multi-template scores.Source: B Theoretical Justification · p. 14

Monotonicity

  • For a fixed calibration-score distribution, the voted-label weight increases with distance from the 0.5 decision boundary.The weight is minimized at ˆvk = 0.5 and maximized at ˆvk ∈ {0, 1}.Source: B Theoretical Justification · p. 14

C.1 Dataset Descriptions

The experiments use five pairwise preference benchmarks, a separate calibration corpus, and a Chat-and-Science training mixture for reward learning. Two held-out benchmarks assess downstream policies without being used to construct the reward or tune aggregation.

Pairwise Preference Benchmarks

  • Five pairwise preference benchmarks evaluate rubric aggregation using gold preference labels for response pairs.RubricBench and ResearchQA have human-sourced rubrics; rubrics for the other three are generated by RubricARM-8B-Rubric.Source: §C.1 · p. 14
  • RubricBench has 1,147 expert-rubric pairs, ResearchQA has 169 scholarly-answer pairs, and RM-Bench yields 387 matched chosen–rejected pairs.RewardBench uses its 456-pair Chat-Hard subset, while RewardBench 2 uses 480 Precise-IF pairs.Source: §C.1 · pp. 14–15

Calibration Data

  • Calibration uses about 2,000 rubric-response-label tuples sampled from 200 domain-stratified prompts in RUBRIC-ARROW-Judge-SFT, disjoint from every evaluation split.The broad coverage is intended to make exchangeability a reasonable approximation; the formal guarantee applies when exchangeability holds.Source: §C.1 · p. 15

RLVR Training Data

  • The RLVR training mixture contains 2,000 RubricHub prompts, split equally between Chat and Science, and uses URW-aggregated verifier judgments as GRPO rewards.The same training corpus is used for baselines to isolate the effect of reward aggregation.Source: §C.1 · p. 15

RLVR Evaluation Benchmarks

  • Policies are evaluated zero-shot on Arena-Hard win rate and GPQA-Diamond accuracy, two held-out benchmarks not used to construct rewards or tune aggregation.Arena-Hard compares against the official gpt-4-0314 response; GPQA-Diamond contains 198 graduate-domain questions.Source: §C.1 · p. 15

C.2 Implementation Details

URW is training-free after calibration: it uses frozen verifiers and deterministic aggregation. The implementation fixes a paraphrased template pool, specifies shared baselines, and evaluates downstream policies with GRPO.

Verifier and aggregation

  • After calibration, URW uses frozen verifiers and deterministic aggregation of binary judgments, without training the aggregation method.The implementation uses DeepSeek V4 Pro, Qwen 3.8 Max, and GLM 5.3 as frozen black-box verifiers.Source: §C.2 · pp. 15–16

Reference-rubric generation

  • For benchmarks without human-sourced rubrics, RubricARM-8B-Rubric generates prompt-specific atomic criteria, and both responses in each pair are scored against that criterion set.The rubric-generation prompt asks for concise, complete, orthogonal, prompt-specific rubrics.Source: §C.2 · pp. 15–16

Verifier and aggregation

  • For each criterion, the verifier is queried with six semantics-preserving judgment templates whose pool is audited and frozen before experiments.Templates are discarded if more than two of 50 audits identify semantics-changing behavior.Source: §C.2 · p. 16

Baselines and hyperparameters

  • Baselines use verifier outputs where their protocols permit; URW adds no tunable hyperparameter, while sampling ensemble, Beta-Binomial, SCP, and Mondrian-SCP use specified alternative weighting rules.Sampling ensemble uses six decodes at T = 0.7; Beta-Binomial uses a Beta(1, 1) prior; SCP variants use α = 0.1, with six bins for Mondrian-SCP.Source: §C.2 · p. 17

Downstream RLVR protocol

  • Downstream evaluation optimizes Qwen 3 8B and GLM 4 9B policies with GRPO for 150 steps on the same 2,000-prompt corpus, changing only reward aggregation across the compared methods.The policies receive no in-domain fine-tuning on Arena-Hard or GPQA-Diamond.Source: §C.2 · pp. 17–18

Evaluation protocols

  • Arena-Hard win rate averages position-swapped pairwise-judge scores, while GPQA-Diamond accuracy is computed from responses with an extractable answer letter.Arena-Hard examples without a parsable verdict from either game are recorded as evaluation errors; failed GPQA answer extractions are also errors.Source: §C.2 · p. 18

D More Experimental Results

The section adds an extended template-count sensitivity analysis on three benchmarks and qualitative case studies of downstream RLVR policies on GPQA-Diamond.

D More Experimental Results

  • The additional results comprise template-count sensitivity on RM-Bench, RewardBench, and RewardBench 2, plus GPQA-Diamond policy case studies.Figure 4 is identified as the template-count sensitivity figure, but the supplied passages do not state its outcomes.Source: D More Experimental Results · p. 18, Figure 4 · p. 19

D.1 More Sensitivity Results

Across RM-Bench, RewardBench, and RewardBench 2, multi-template aggregation generally improves accuracy over Uniform aggregation, with most gains appearing at small template counts and performance then largely plateauing.

D.1 More Sensitivity Results

  • Multi-template aggregation generally achieves higher accuracy than Uniform aggregation across RM-Bench, RewardBench, and RewardBench 2 and the evaluated verifier backbones.The reported pattern extends beyond a particular benchmark or model family.Source: §D.1 · p. 19
  • Most performance gains emerge with a small template pool, after which accuracy curves largely plateau or vary only modestly.URW does not require delicate tuning of template count N to obtain its benefit.Source: §D.1 · p. 19

D.2 Case Study on Downstream RLVR

Qualitative GPQA-Diamond case studies compare policies trained with URW against Uniform aggregation and uncertainty-aware baselines. URW-trained policies answer both cases correctly, while the other shown policies answer their respective cases incorrectly; these examples are not a general performance guarantee.

D.2 Case Study on Downstream RLVR

  • Only URW-trained Qwen 3 8B and GLM 4 9B policies produce the correct answer in both GPQA-Diamond cases; all shown alternatives answer their respective cases incorrectly.Tables 7 and 8 present the Qwen 3 8B and GLM 4 9B cases, respectively.Source: §D.2 · p. 19, Table 7 · p. 20, Table 8 · p. 22
  • The case studies compare Uniform aggregation and uncertainty-aware baselines against URW on questions where the methods produce different final answers.The analysis is conducted on GPQA-Diamond.Source: §D.2 · p. 19
  • These qualitative examples do not establish a general performance guarantee, but are consistent with the aggregate GPQA-Diamond results.The authors suggest that the examples illustrate how calibrated criterion-level weighting can support downstream scientific reasoning.Source: §D.2 · p. 19

Related papers

Loading 2610.21214v1…