PaperlayerView exact revision on arXiv

Source-linked AI summary

Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation

Haocheng Yang, Licheng Pan, Yiming Dong, Xiaoxi Li, Yuan Lu, Zhenhao Chen, Jiajun Fan, Shuai Shao, Chunyuan Zheng, Hao Wang

arXiv:2610.19915v1cs.CL

TL;DR

Uniformly weighting rubric criteria can produce misleading scores when criteria differ in importance or overlap. ICW derives weights from pairwise LLM judgments, checks and revises inconsistent comparisons, and consistently outperforms uniform and competitive weighting methods while yielding better downstream RL policy models.

  • Problem

    Uniform weighting treats essential, minor, and overlapping rubric criteria alike, which can produce misleading scores that favor lower-quality responses.

    Source: §1 · p. 1, §3.1 · p. 3

  • Method

    ICW derives criterion weights from pairwise LLM importance judgments, checks and revises inconsistent comparisons, and obtains weights from the comparison matrix’s principal eigenvector.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    ICW consistently outperforms uniform and competitive weighting methods, improving pairwise preference accuracy over uniform weighting by up to 5.79%.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    ICW’s improved evaluation scores translate into better policy models in downstream reinforcement learning.

    Source: §1 · p. 2, §5 · p. 9

    ICW’s pairwise comparisons make query counts grow quadratically with criterion count, and specialized domains such as healthcare may exceed the weighting model’s knowledge.

    Source: §5 · p. 10

Abstract

from arXiv · show

Rubric-based evaluation assesses the responses of large language models (LLMs) by aggregating the verification results of explicit criteria, yet not all criteria matter equally. However, existing methods commonly aggregate the verification results with uniform weighting, which assigns minor and overlapping criteria the same weight as essential ones and thus produces misleading evaluation scores. Although some methods prompt an LLM to directly assign weights, direct weighting requires a holistic judgment over all criteria that LLMs struggle to make reliably. To address these limitations, we propose Importance-Aware Criteria Weighting (ICW), which derives the weights of criteria from pairwise importance comparisons made by an LLM. Specifically, ICW transforms the holistic judgment over all criteria into a series of pairwise importance comparisons, checks and revises inconsistent comparisons, and derives the weights from the principal eigenvector of the comparison matrix. Extensive experiments demonstrate that ICW consistently outperforms uniform weighting and competitive weighting methods, and translates into better policy models in downstream reinforcement learning.

1 INTRODUCTION

Rubric-based evaluation aggregates criterion-level verifier results, but uniform weighting treats criteria as equally important despite differences in importance and overlap. ICW derives weights from pairwise importance comparisons and revises inconsistent judgments; experiments report improved evaluation and downstream RL outcomes.

1 INTRODUCTION

  • Prior work improves rubric evaluation through rubric generation and refinement, or uses evaluation scores as reward signals for open-ended-task RL.Rubrics may be expert-written or LLM-generated and refined to distinguish responses.Source: §1 · p. 1
  • Uniform weighting can give minor or overlapping criteria the same influence as essential ones, producing misleading scores that may favor lower-quality responses.Direct numerical weighting also requires joint judgments over the rubric set, which studies of LLM evaluation associate with anchoring and inconsistent ratings.Source: §1 · pp. 1–2
  • ICW derives criterion weights from pairwise LLM importance comparisons, checks and revises inconsistencies, and uses the comparison matrix’s principal eigenvector.Each comparison asks which of two criteria matters more for the prompt, avoiding a single holistic judgment over the full set.Source: §1 · p. 2
  • Across six pairwise preference benchmarks, ICW outperforms uniform and competitive weighting methods and yields better policy models in downstream RL.The contribution summary reports comprehensive experiments across six benchmarks.Source: §1 · p. 2

2 PRELIMINARIES

Rubric-based evaluation combines weighted criterion-verifier results into a response score. The paper frames rubric weighting as estimating weights from the prompt and fixed rubric so the score ranks candidate responses consistently with human preference.

2.1 RUBRIC-BASED EVALUATION

  • Rubric-based evaluation aggregates weighted LLM-verifier judgments of K criteria into a response-quality score that can rank candidate responses or serve as an RL reward.Each criterion describes a property of an ideal response, and the verifier indicates whether the response satisfies it.Source: §2.1 · p. 2

2.2 PROBLEM DEFINITION

  • The weighting problem estimates criterion weights from the user prompt and fixed rubric so the resulting score ranks candidate responses consistently with human preference.The weights are shared across candidate responses for the prompt and rubric.Source: §2.2 · p. 2

3 METHODOLOGY

ICW addresses misleading scores from uniform rubric weighting by deriving criterion weights from pairwise importance comparisons. It checks and revises inconsistent comparisons before using the principal eigenvector to obtain weights.

3.1 MOTIVATION

  • Uniform weighting can let minor or overlapping criteria outweigh essential ones, giving lower-quality responses higher evaluation scores.Figure 1 illustrates both cases: stylistic criteria outweigh factual correctness, and overlapping criteria repeatedly credit one recommendation.Source: §3.1 · p. 3, Figure 1 · p. 3

3.2 PAIRWISE IMPORTANCE COMPARISON

  • ICW compares each criterion pair in both display orders and encodes an agreed winner with a fixed ratio α, treating ties or disagreement as neutral.The comparison matrix is positive and reciprocal, summarizing relative criterion importance for the prompt.Source: §3.2 · p. 4

3.3 IMPORTANCE WEIGHTING VIA PRINCIPAL EIGENVECTOR

  • ICW derives criterion weights from the comparison matrix's principal eigenvector, which is well defined even when comparisons are inconsistent.For consistent comparisons, the pairwise ratios match the ratios between the corresponding weights.Source: §3.3 · pp. 4–5

3.4 CONSISTENCY CHECK AND REVISION

  • ICW measures inconsistency with CR, accepts matrices at or below threshold τ, and revises comparisons only when a revision strictly lowers CR.Re-comparisons use the prior round's judgments and reasons and stop when the threshold is met or after at most T rounds.Source: §3.4 · p. 5

3.5 THE WORKFLOW OF ICW

  • The workflow queries each pair independently, allowing parallel calls, and computes weights once per prompt and rubric set for sharing across candidate responses.Each round uses K(K −1) queries, and the cost does not grow with the number of candidate responses.Source: §3.5 · pp. 5–6

4 EXPERIMENTS

The experiments evaluate ICW on six preference benchmarks, test its components and robustness, and assess downstream reinforcement learning. ICW outperforms all baselines across the benchmarks and achieves the best overall downstream RL performance on nearly all reported benchmarks across policy models.

4.1 EXPERIMENTAL SETUP

  • The setup uses six pairwise-preference benchmarks, shared rubric sets and verification results, and pairwise preference accuracy as its primary metric.The benchmarks are RubricBench, ResearchQA, RM-Chat, Chat-Hard, JudgeBench, and Precise-IF; relative improvement is reported over Uniform.Source: §4.1 · pp. 6–7

4.2 OVERALL PERFORMANCE (RQ1)

  • ICW achieves state-of-the-art accuracy across all six benchmarks, outperforming every baseline and improving over Uniform by up to 5.79%.The comparison includes uniform, direct, and indirect weighting methods.Source: §4.2 · p. 7, Table 1 · p. 7

4.3 ABLATION STUDY (RQ2)

  • Both ICW components contribute: pairwise importance comparisons provide the dominant gain, while consistency checking and revision add complementary gains.Comparing all pairs in one query lowers JudgeBench accuracy from 0.783 to 0.758; omitting revision lowers Chat-Hard accuracy from 0.818 to 0.804.Source: Table 2 · p. 7, §4.3 · p. 8

4.4 GENERALIZATION STUDIES (RQ3)

  • ICW generally outperforms Uniform across five verifiers and consistently outperforms it with all five tested weighting models.The experiments include models from Gemini, GPT, DeepSeek, and GLM families.Source: §4.4 · p. 8, Table 3 · p. 8, Table 4 · p. 8

4.5 HYPERPARAMETER SENSITIVITY STUDY (RQ4)

  • With any tested α ≥2, ICW beats RRD-WU in average accuracy across four benchmarks; accuracy remains stable as α increases, while lower τ generally improves it.Lower τ triggers more revisions; at large τ, ICW approaches its no-revision variant.Source: §4.5 · pp. 8–9, Figure 3 · p. 8

4.6 DOWNSTREAM RL PERFORMANCE (RQ5)

  • ICW achieves the best overall downstream RL results on nearly all benchmarks across policy models.For Qwen2.5-3B-Instruct, AlpacaEval LC win rate improves by 40.87% over Base Model, compared with 14.12% for Uniform.Source: §4.6 · p. 9, Table 5 · p. 9

5 CONCLUSION

ICW replaces uniform rubric weighting with weights derived from pairwise importance comparisons, then checks and revises inconsistent comparisons. The paper reports improved evaluation accuracy and downstream policy-model results, while identifying query cost and specialized-domain knowledge as limitations.

5 CONCLUSION

  • ICW consistently outperforms uniform and competitive weighting, improving pairwise preference accuracy over uniform by up to 5.79%.For Qwen2.5-3B-Instruct, the AlpacaEval LC win rate improves 40.87% over Base Model, compared with 14.12% for uniform weighting.Source: §5 · pp. 9–10
  • ICW derives criterion weights from pairwise LLM importance comparisons and checks and revises inconsistent comparisons.The paper reports that the principal eigenvector of the comparison matrix supplies the weights.Source: §5 · p. 9
  • ICW compares every criterion pair, making query counts grow quadratically; in specialized domains such as healthcare, the weighting model may lack needed knowledge.The authors identify informative-pair selection and expert-specified comparisons as directions for future work.Source: §5 · p. 10

A RELATED WORK

  • Prior rubric work covers rubric construction, coverage, and downstream use, while this paper focuses on how criterion weights are obtained.Existing automatic weighting approaches include direct LLM weighting, pairwise comparisons, and downweighting correlated criteria from sampled responses.Source: A RELATED WORK · p. 15

B THEORETICAL JUSTIFICATION

  • For the theoretical justification, the paper states that the principal eigenvector preserves the order of transitive judgments.The supplied passage points to proofs of Propositions 1 and 2 and the derivation of Eq. (4).Source: B THEORETICAL JUSTIFICATION · p. 15

B.1 PROOF OF PROPOSITION 1

The proofs connect consistent pairwise comparisons to normalized criterion weights and the principal eigenvector, establish an inconsistency bound, and show that transitive judgments retain their ordering in the resulting weights.

B.1 PROOF OF PROPOSITION 1

  • For a consistent comparison matrix, the proof constructs normalized positive weights from its first column and shows that the weights satisfy a_ij = w_i/w_j.The resulting vector is the principal eigenvector with eigenvalue K.Source: §B.1 · p. 15

B.2 PROOF OF PROPOSITION 2

  • The proof of Proposition 2 establishes λmax ≥ K, with equality exactly when the comparison matrix is consistent.It also derives Eq. (4) using the K(K −1)/2 criterion pairs.Source: §B.2 · p. 16

B.3 ORDER PRESERVATION OF THE PRINCIPAL EIGENVECTOR

  • For transitive judgments, the principal eigenvector preserves strict and tied importance order for any α > 1, even when the encoded matrix is inconsistent.The proof compares rows of Aw = λmaxw; the passage notes inconsistency can arise when importance levels have more than two distinct values.Source: §B.3 · p. 16

C MORE EXPERIMENTAL RESULTS

The supplementary section reports extensions of the generalization and sensitivity studies, followed by ICW's token cost.

C MORE EXPERIMENTAL RESULTS

C.1 MORE GENERALIZATION RESULTS

The extended results report that ICW generally beats uniform weighting across verifier choices and consistently beats it across weighting-model choices on three additional benchmarks.

C.1 MORE GENERALIZATION RESULTS

  • With GPT-4.1 as weighting model, ICW generally outperforms Uniform across five verifiers on RubricBench, ResearchQA, and JudgeBench.The table caption identifies results across verifiers on the other benchmarks.Source: §C.1 · p. 16, Table 6 · p. 17
  • With Gemini-3.1-Flash-Lite as verifier, ICW consistently outperforms Uniform across all five weighting models on the three benchmarks.The paper says this indicates pairwise importance comparison does not rely on a particular weighting model.Source: §C.1 · p. 16
  • The extended generalization study covers RubricBench, ResearchQA, and JudgeBench.Source: §C.1 · p. 16

C.2 MORE SENSITIVITY RESULTS

Across Gemini-3.5-Flash and GLM-5.1, ICW outperforms uniform weighting on all four benchmarks for every tested α ≥ 2. Accuracy remains stable as α increases beyond 2.

C.2 MORE SENSITIVITY RESULTS

  • Table 7 reports results across weighting models on the other benchmarks.Source: Table 7 · p. 17
  • For both weighting models, ICW outperforms uniform weighting (α = 1) on all four benchmarks for every tested α ≥ 2.This indicates that the benefit of pairwise importance comparison does not depend on a particular weighting model.Source: §C.2 · p. 17
  • For both weighting models, accuracy remains stable as α increases beyond 2.This extends the robustness to α observed with GPT-4.1 to Gemini-3.5-Flash and GLM-5.1.Source: §C.2 · p. 17

C.3 COST ANALYSIS

The cost analysis reports ICW's per-sample token use across weighting models, with input and output tokens shown in thousands.

C.3 COST ANALYSIS

  • Figure 4 presents sensitivity to the fixed ratio α for Gemini-3.5-Flash and GLM-5.1 as weighting models.Source: Figure 4 · p. 17
  • Table 8 reports the average input and output tokens consumed by the weighting model per sample.The values are in thousands of tokens, and output counts include reasoning tokens.Source: §C.3 · p. 17

D.1 DATASET DESCRIPTIONS

The experiments use pairwise preference benchmarks to evaluate rubric weighting and separate datasets and benchmarks for downstream reinforcement learning. The pairwise benchmarks pair user prompts and candidate responses with preference labels.

Pairwise Preference Benchmarks

  • The pairwise preference benchmarks provide user prompts, candidate-response pairs, and preference labels.Source: §D.1 · p. 18
  • RubricBench targets nuanced, difficult samples with expert-written rubrics derived from each instruction.Its samples include misleading surface features.Source: §D.1 · p. 18
  • ResearchQA pairs scholarly questions distilled from survey articles with query-specific rubric items and human preference annotations.Source: §D.1 · p. 18
  • JudgeBench covers challenging knowledge, reasoning, math, and coding response pairs, with labels reflecting factual and logical correctness.Source: §D.1 · p. 18
  • The reward-model benchmarks include RM-Bench's Chat subset, RewardBench's Chat-Hard subset, and RewardBench 2's Precise-IF subset.These subsets address subtle content differences and style bias, challenging chat pairs, and precise instruction constraints, respectively.Source: §D.1 · p. 18
  • Rubric sets for JudgeBench, RM-Chat, Chat-Hard, and Precise-IF come from the RubricARM-8B-Rubric generator of Rubric-ARM (Xu et al., 2026)8.Source: §D.1 · p. 18

Downstream RL Data and Benchmarks

  • WildChat user prompts train the RL policies, and Gemini-3.5-Flash generates a rubric set for each prompt; BiGGen Bench evaluates nine capabilities on a 0–100 scale.AlpacaEval 2.0 reports WR and length-controlled win rate (LC); Arena-Hard reports WR against a baseline and style-controlled win rate (SC) on Arena-Hard-Auto v0.1.Source: §D.1 · p. 18

D.2 IMPLEMENTATION DETAILS

The implementation details specify consistency-check settings, downstream RL training and evaluation, and prompts for weighting and verification. The weighting-model prompts compare criterion pairs and can revise inconsistent judgments.

D.2 IMPLEMENTATION DETAILS

  • Consistency ratios use Saaty's random indices for K ≤ 15 and RI15 for K > 15.Source: §D.2 · p. 18
  • For downstream RL, GRPO trains Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-4B from WildChat prompts, with GPT-4.1 weighting and GPT-4.1-mini verification.All methods share rubric sets; evaluation uses GPT-4.1 as judge on all benchmarks.Source: §D.2 · p. 19
  • The weighting-model prompt defines importance as how much failing a criterion should reduce credit and compares two criteria using the prompt and fixed rubric set.Prompt D.2 revisits a highlighted pair against the prompt and complete rubric set, retaining supported judgments or revising unsupported ones without inventing a preference.Source: §D.2 · pp. 19–20
  • Pairwise comparisons instruct the model to list explicit prompt constraints first, avoid unsupported preferences, account for duplicates and vague criteria, and remain invariant to display order.The output is JSON with a winner or neutral judgment and a reason of no more than 25 words.Source: §D.2 · p. 19
  • Consistency review distinguishes agreement, both-neutral, one-order-neutral, and opposite-winner cases; order instability does not establish equal importance.Even both-neutral judgments may reflect insufficient evidence rather than established equality.Source: §D.2 · pp. 19–20
  • The verifier evaluates one criterion locally, marking it satisfied only when clearly and completely met, while preserving quantifiers and checking meaning rather than keyword overlap.It returns a strict JSON judgment with a concise evidence-based reason.Source: §D.2 · p. 20

Related papers

Loading 2610.19915v1…