PaperlayerView exact revision on arXiv

Source-linked AI summary

MatrixReward: Reward from Rubric Matrix for Open-Ended Generation

Zihan Shen, Qi Liu, Zixuan Yang, Yiqun Chen, Chenglong Zhao, Xiaozhao Wang, Lei He

arXiv:2610.00389v1cs.LGcs.AIstat.ML

TL;DR

Open-ended generation lacks standard answers, and pointwise rubric scores can provide limited evidence about relative response quality. MatrixReward derives rewards from pairwise rubric win-rate matrices, data-dependent and prior rubric weights, and distances to ideal profiles. On four benchmarks, Qwen3-8B averages 63.02, exceeding the strongest baselines by 1.23, 6.93, and 5.60 points.

  • Problem

    Open-ended generation lacks standard answers, while pointwise rubric scores provide limited information about relative quality and weighted rubric judgments may mask response differences.

    Source: §1 · pp. 1–2

  • Method

    MatrixReward forms a rollout-by-rubric win-rate matrix from pairwise comparisons, combines matrix-derived weights with rubric importance priors, and uses distances to ideal profiles to construct rewards.

    Source: §1 · p. 2

  • Results

    63.02 average across four benchmarks; MatrixReward exceeds the strongest pairwise, pointwise, and weight-aware baselines by 1.23, 6.93, and 5.60 points, respectively.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Ablations show that calibrating rubric weights improves downstream performance and that the calibration rule matters.

    Source: §5 · p. 9

    Policies optimized with LLM judges may inherit systematic judge biases or favor superficial properties over factuality and user utility, warranting independent safety evaluation, factuality checks, and human review in deployment.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Open-ended query generation lacks standard answers, thus necessitating an effective reward mechanism. Pointwise scoring rubrics provide limited information about the relative quality of sample answers under the same prompt; merging multiple rubric judgments into a single score may also mask the differences between these answers. We propose MatrixReward, which constructs rewards from a rollout-by-rubric win-rate matrix obtained by comparing every pair of sampled responses under each rubric. The spread of each matrix column captures how strongly that rubric distinguishes the current rollouts, while correlations between columns reveal rubric repetition; together, these statistics yield data-dependent rubric weights. We combine these weights with the prior weights of rubrics. After column normalization and weighting, the observed per-rubric maxima and minima define positive and negative ideal profiles. Each rollout's distances to these two ideals determine its relative-closeness quality reward. Evaluated using Qwen3-8B on four open-ended query-answering benchmarks, MatrixReward achieves an average score of 63.02, outperforming the strongest baseline by approximately 2.0%. These results support the idea that matrices derived from relative comparisons can be used to construct rewards more reasonably for open-ended generative reinforcement learning.

1 INTRODUCTION

Open-ended generation needs rewards that distinguish qualities among responses to the same prompt, but pointwise scores and fixed rubric weights may obscure relative quality. MatrixReward uses rubric-wise pairwise win rates and ideal-profile distances to construct rewards.

1 INTRODUCTION

  • Open-ended responses can be fluent yet uninformative or detailed yet poorly grounded, so rewards must distinguish qualities among responses to the same prompt.Unlike verifiable tasks, open-ended generation has no concrete correct-answer target.Source: §1 · p. 1
  • Pointwise rubric scores lack a stable anchor without reference answers, and the same value may reflect different standards across queries or evaluators.Rubric weights set in advance do not ensure rollout differentiation; correlated rubrics can duplicate scoring, while weighted averages can cancel differences across dimensions.Source: §1 · pp. 1–2
  • MatrixReward compares every pair of rollouts under each query-specific rubric, then uses the resulting profiles, query-specific weights, and positive and negative ideals to construct rewards.The matrix preserves separate rubric-wise win-rate profiles, exposing which rubrics distinguish the current rollouts and which overlap.Source: §1 · p. 2
  • 63.02 average across four benchmarks; MatrixReward exceeds the strongest pairwise, pointwise, and weight-aware baselines by 1.23, 6.93, and 5.60 points, respectively.Ablations examine weight extraction and fusion, ideal-profile selection, distance metrics, and judging cost.Source: §1 · p. 2

2 RELATED WORK

Related work addresses rubric design, pairwise comparisons, adaptive weighting, and aggregation of multidimensional feedback. These lines of work differ in how they evaluate responses, adapt weights, and construct rewards.

2 RELATED WORK

  • Rubrics as Rewards uses instance-specific rubrics with weighted rubric-level assessments and rubric-conditioned holistic judgments; Chasing the Tail diversifies rubrics using examples, while OpenRubrics derives them from preference data.These approaches improve what is judged, while pointwise rubric scores versus pairwise comparisons remains consequential for reward construction.Source: §2 · p. 2
  • Tournament-GRPO, ArenaRL, and OpenRS use rubric-guided or adaptive pairwise comparisons for RL supervision, while MAPLE estimates evaluator reliability and aspect importance for assessment.Their comparison procedures and use of rubric information vary.Source: §2 · p. 3
  • POW3R, SAW, SA-MRPO, and AMRP adapt reward weights using rollout contrast or other training-state statistics, but differ in whether they account for rubric redundancy within a response group.The methods use different signals, including reward variation, saturated objectives, shortfall, volatility, and stalled progress.Source: §2 · p. 3
  • GDPO, GD2PO, OpenRS, and AMRP address aggregation through separate normalization, conflict handling, preference aggregation, or analysis of fixed scalar projections.The approaches target different issues in turning multidimensional feedback into rewards.Source: §2 · p. 3

3 METHOD

MatrixReward builds a query-specific reward from pairwise rubric judgments, matrix-derived and prior rubric weights, and each response’s distances to observed ideal profiles.

3 METHOD

  • MatrixReward constructs a rollout-by-rubric matrix of win rates, combines data-driven rubric weights with rubric-creation importance weights, and derives rewards from distances to observed ideal profiles.Its workflow is applied separately to each query and its sampled response group.Source: §3 · p. 3

3.1 CONSTRUCTING THE RUBRIC-WISE COMPARISON MATRIX

  • For each query, every pair of responses is compared once under each rubric; wins, losses, and ties count as 1, 0, and 1/2, respectively, and outcomes are averaged over other responses into rubric-specific win rates.Source: §3.1 · p. 4

3.2 CALIBRATING MATRIX-DERIVED AND PRIOR WEIGHTS

  • Matrix-derived rubric weights combine column variation with a discount for correlation, then mix those weights with query-specific importance priors using a group-specific least-squares combination.Constant columns receive zero data-derived weight; when the combination system is singular or nonfinite, the implementation uses a normalized geometric mixture of the two weight sources.Source: §3.2 · p. 4

3.3 CONSTRUCTING A REWARD FROM MULTIDIMENSIONAL PROFILES

  • Each rubric column is normalized and weighted; its observed maxima and minima define positive and negative ideal profiles, and relative distances to the ideals determine the quality reward.Responses closer to the positive ideal and farther from the negative ideal receive larger quality rewards.Source: §3.3 · p. 5
  • A fixed format-validity reward is added to quality reward, and GRPO normalizes the resulting rewards within the query’s response group to form response-level advantages.The advantages are assigned to response tokens and used in the clipped GRPO policy objective.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Across four open-ended benchmarks, MatrixReward outperforms comparison baselines and its component ablations. Experiments also examine rubric-weight extraction, ideal-profile reward construction, weight fusion, and judging cost.

4.1 EXPERIMENTAL SETUP

  • Experiments train Qwen3-8B with GRPO on 2,560 UltraFeedback queries, using eight rollouts per query and reporting results after three epochs (480 steps).Four benchmarks are evaluated on a 0–100 scale, with Avg denoting their mean.Source: §4.1 · p. 6

4.2 COMPARISON WITH BASELINES

  • 63.02 four-benchmark average beats RRC-SCR by 1.23, DAPO by 6.93, and SAW by 5.60 points; MatrixReward leads on three of four benchmarks.ArenaRL scores 0.04 points higher on WildBench; the implicit variant achieves a 62.23 average.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 EXTRACTING RUBRIC WEIGHTS FROM THE MATRIX

  • MatrixReward averages 63.02 across four benchmarks, versus 60.24 for the prior, 61.46 for Entropy, and 62.16 for SVD.Entropy uses within-column dispersion but omits cross-rubric correlation; SVD retains shared variation among correlated rubrics, with information loss described as limited in this setting.Source: §4.3 · pp. 6–7, Table 2 · p. 7

4.4 REWARD AGGREGATION AND DISTANCE METRIC

  • 63.02 average for MatrixReward exceeds 59.83 for weighted aggregation, 60.81 for theoretical ideals, and 61.05 for Mahalanobis distance.The comparison retains the same method except for the reward aggregation or distance choice being varied.Source: Table 3 · p. 7, §4.4 · p. 7
  • Across 800 groups, empirical ideals yield broader rewards than theoretical ideals, while Euclidean rewards vary more than Mahalanobis rewards (SD 0.26 vs 0.20).Empirical rewards average 0.51 while theoretical rewards cluster near 0.33; in one example, MatrixReward reverses weighted aggregation’s y1/y3 ranking by comparing distances to both ideals.Source: Figure 3 · p. 8, §4.4 · p. 8, Figure 4 · p. 8

4.5 SENSITIVITY TO WEIGHT FUSION

  • The group-dependent weight combination averages 63.02, compared with 62.04 for additive fusion at α = 0.5 and 59.98 for multiplicative fusion.The additive average rises from 61.47 at α = 0.1 to 62.04 at α = 0.5, with which benchmark benefits most varying by α.Source: Table 4 · p. 9, §4.5 · p. 9

4.6 JUDGING COST AND OUTCOME

  • Implicit judging averages 62.23 at 0.52 billion judge tokens, 0.79 points below explicit judging’s 63.02 at 2.00 billion tokens.Implicit judging uses 74% fewer tokens and exceeds RRC-SCR by 0.44 points while using 0.06 billion fewer tokens.Source: Table 5 · p. 9, §4.6 · p. 9

5 CONCLUSION

MatrixReward builds relative rewards from rubric-wise pairwise comparisons and reports improved benchmark performance for Qwen3-8B. The authors also discuss automated-judge risks and reproducibility materials.

5 CONCLUSION

  • 63.02 average across four benchmarks exceeds the strongest pairwise, pointwise, and weight-aware baselines by 1.23, 6.93, and 5.60 points, respectively.The ablations also show that rubric-weight calibration improves downstream performance and that calibration rules, reference contours, and distance metrics matter.Source: §5 · p. 9
  • MatrixReward converts rubric-wise pairwise judgments into a rollout-by-rubric win-rate matrix, calibrates query-specific weights, and derives rewards from distances to observed ideal and anti-ideal profiles.Source: §5 · p. 9
  • The paper used generative AI tools only for language editing, with authors reviewing revisions and taking responsibility for the final content.Source: AI USE STATEMENT · p. 10
  • Policies optimized with LLM judges may inherit systematic biases or favor superficial properties over factuality and user utility.For high-stakes deployment, the authors recommend independent safety evaluation, factuality checks, and appropriate human review.Source: ETHICS STATEMENT · p. 10
  • The appendix provides prompts, implementation hyperparameters, Qwen2.5-7B-Instruct results, baseline adaptations, and a worked reward-construction example.Source: REPRODUCIBILITY STATEMENT · p. 10

A PROMPT TEMPLATES

The appendix provides rubric-generation prompts and two pairwise-judging formats: one evaluates a rubric at a time, while the other returns judgments across rubrics in one call.

A PROMPT TEMPLATES

  • The appendix contrasts an explicit judge that receives one rubric at a time with an implicit judge that returns verdicts across rubrics in one call.Source: A PROMPT TEMPLATES · p. 12

A.1 RUBRIC-GENERATION PROMPT

  • Rubric generation uses four answers ordered by overall quality as guidance, while grounding 3–5 concise, query-specific criteria in the query.The prompt advises selecting applicable dimensions rather than mechanically including every suggested quality dimension.Source: §A.1 · p. 12
  • Generated rubrics must apply to alternative valid answers, avoid duplicates and unsupported requirements, and receive integer importance weights from 1 to 3.Source: §A.1 · p. 13
  • An alternative rubric-generation prompt requests 3 to 5 distinct, query-specific criteria with integer weights from 1 to 3 in a constrained JSON object.Its input includes four candidate answers in descending order of existing overall scores.Source: §A.1 · p. 13

A.2 EXPLICIT RUBRIC-WISE PAIRWISE JUDGE PROMPT

  • The explicit judge compares two responses on one dimension, returns A, B, or TIE, and has no separate system prompt.It is instructed not to treat response order, length, or formatting as evidence unless the rubric concerns them.Source: §A.2 · pp. 13–14
  • The explicit judge's response format assigns half a point to each response for a tie and averages identity-aligned outcomes if both display orders are judged.Invalid responses are retried up to three times; a still-failed required request gives the group a neutral common reward.Source: §A.2 · p. 14

A.3 IMPLICIT RUBRIC-WISE PAIRWISE JUDGE PROMPT

  • The implicit judge returns one A, B, or TIE verdict per rubric in rubric order, with an explicit tie option and no separate system prompt.Source: §A.2 · p. 13, §A.3 · p. 14

B IMPLEMENTATION DETAILS

The implementation details specify shared experiment settings and provide a hyperparameter table.

B IMPLEMENTATION DETAILS

C QWEN2.5-7B-INSTRUCT RESULTS

The appendix reports Qwen2.5-7B-Instruct results using the same training strategy.

C QWEN2.5-7B-INSTRUCT RESULTS

D BASELINE REPRODUCTION AND ADAPTATION

The baselines retain their defining reward or advantage transformations while using judgments on the same open-ended task. They include pointwise, pairwise, tournament, and weight-adaptive approaches.

D BASELINE REPRODUCTION AND ADAPTATION

  • GRPO and Dr. GRPO use independent holistic pointwise scores, with Dr. GRPO additionally removing group-standard-deviation scaling and the actor KL term.Dr. GRPO also uses a fixed-horizon token-sum loss.Source: D BASELINE REPRODUCTION AND ADAPTATION · p. 16
  • DAPO pairs holistic pointwise judging with Clip-Higher, dynamic sampling, token-level loss, and overlong-response shaping.The open-ended judge score replaces DAPO’s original verifiable reward.Source: D BASELINE REPRODUCTION AND ADAPTATION · p. 16
  • GDPO normalizes rubric-wise rewards within the group, GD2PO filters cross-dimension sign conflicts, and GOPO converts aggregated rubric scores to ordinal rewards.GD2PO scales the retained query signal, while GOPO uses within-group ordering.Source: D BASELINE REPRODUCTION AND ADAPTATION · p. 16
  • RRC-SCR uses opponent-averaged holistic pairwise win rates as scalar rewards, whereas Pairwise Rank keeps only their reversed average-rank ordering.Pair outcomes are scored as one for a win, one-half for a tie, and zero for a loss.Source: D BASELINE REPRODUCTION AND ADAPTATION · p. 16
  • ArenaRL judges each unordered pair in both candidate orders and maps competition ranks to rewards, while Tournament-GRPO advances judged winners through shuffled elimination brackets.Tournament-GRPO accumulates tournament scores and normalizes group rewards; repeated bracket count is run-specific.Source: D BASELINE REPRODUCTION AND ADAPTATION · p. 16
  • SAW, POW3R, and AMRP adapt rubric weights using batch or group statistics and moving-state mechanisms, with weights computed within each query group.The within-query adaptation addresses query-specific rubric columns having no shared meaning across queries.Source: D BASELINE REPRODUCTION AND ADAPTATION · p. 16

E WORKED EXAMPLE OF MATRIXREWARD

The worked poetry example applies MatrixReward to eight sampled responses, deriving rubric weights and ideal profiles from pairwise win rates before computing response rewards and group-normalized advantages.

E WORKED EXAMPLE OF MATRIXREWARD

F ALTERNATIVE MATRIX-WEIGHT EXTRACTORS

The alternative extractors derive rubric weights from the same group-level win-rate matrix using either normalized-column entropy or an SVD-based variance representation.

F ALTERNATIVE MATRIX-WEIGHT EXTRACTORS

  • The entropy extractor assigns weights proportional to one minus each column’s entropy after min–max normalization, favoring less-uniform normalized win rates.It forms q_ik from normalized column values and computes entropy relative to log G.Source: F ALTERNATIVE MATRIX-WEIGHT EXTRACTORS · p. 18
  • The SVD extractor standardizes nonconstant matrix columns, computes eigenpairs of Z⊤Z/G, and retains the fewest components explaining at least 85% of variance.Weights are set from the retained components.Source: F ALTERNATIVE MATRIX-WEIGHT EXTRACTORS · p. 18
  • Both extractors normalize weights to sum to one, assign constant columns zero weight, and use uniform weights when the entire matrix is constant.Both operate on the same group-level win-rate matrix as the main method.Source: F ALTERNATIVE MATRIX-WEIGHT EXTRACTORS · p. 18

Related papers

Loading 2610.00389v1…