PaperlayerView exact revision on arXiv

Source-linked AI summary

SetGRPO: Calibrating Grader-Induced Policy Optimization

Jiale Huang, Zixu Li, Zhiwei Chen, Zhiheng Fu, Jinhe Bi, Yupeng Hu

arXiv:2610.36807v1cs.LGcs.AI

TL;DR

Grader randomness can alter GRPO updates in ways that score discrepancies alone do not characterize, because group-normalized coefficients interact with current-policy gradients and selective execution changes accepted groups. SetGRPO calibrates joint coefficient uncertainty, recovers updates using policy geometry, and gates updates through risk calibration. Across ten benchmarks spanning three domains, it achieves the highest macro average among compared methods on both evaluated model families.

  • Problem

    Prior rubric-based GRPO work does not characterize how stochastic grader scores propagate through shared group normalization and response-level gradients to alter local policy updates.

    Source: §1 · p. 2

  • Method

    SetGRPO calibrates a joint uncertainty set for GRPO coefficients, uses current-policy update geometry for safe recovery, and risk-calibrates a global threshold for accepted updates.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    SetGRPO achieves the highest Macro Avg. on Qwen3.5-4B and Qwen3.5-9B, reaching 60.26 and 63.51, respectively, and is reported as best among compared methods on both model families.

    Source: §1 · p. 2, §5.2 · p. 9

  • Takeaways & Limitations

    SetGRPO’s joint coefficient modeling and selective screening are reported to improve performance across backbones and task domains under rubric score discrepancies.

    Source: §1 · p. 2, §5.2 · p. 9

    Its finite-sample guarantee concerns only conditional direction-reversal risk for accepted updates under exchangeability, not privacy, fairness, factuality, or harmlessness.

    Source: §8 · p. 22

Abstract

from arXiv · show

Rubric-based reinforcement learning uses fine-grained grader feedback to train large language models on open-ended tasks, but grader randomness can alter GRPO updates in ways that score discrepancies alone cannot characterize. Because GRPO jointly normalizes rewards within each response group and combines the resulting training coefficients with response-level gradients, a scoring discrepancy can shift the coefficients of the entire group and induce policy-dependent update changes. Selective execution further affects the direction-reversal risk of retained updates. To address this, we propose Set-Valued Group Relative Policy Optimization (SetGRPO). It comprises three modules: (a) Joint Reward Set Calibration constructs a joint uncertainty set for the training coefficients; (b) Geometry-Guided Safe Recovery maps candidate coefficients into the current policy's local update geometry and determines the largest safe recovery ratio for each response group; and (c) Selective Risk-Gated Optimization selects a global threshold through independent risk calibration to control the conditional direction-reversal risk of accepted updates while preserving the original GRPO objective. Under exchangeability, SetGRPO provides finite-sample marginal coverage for the joint coefficient set and simultaneous risk control across candidate thresholds. On ten open-ended benchmarks spanning three domains, SetGRPO achieves the highest macro average among all compared training methods on both Qwen3.5 and Llama3.2 model families.

1. Introduction

Rubric-based GRPO uses fine-grained grader scores, but prior work does not characterize how score discrepancies propagate through group normalization and policy gradients to change updates. SetGRPO models coefficient uncertainty, update geometry, and accepted-update risk; it achieves the best performance among compared methods on both evaluated model families.

1. Introduction

  • Prior work largely studies training-signal construction or assessment, rather than how grader randomness propagates through GRPO normalization and gradients to alter policy updates.GRPO derives group coefficients from shared reward statistics and combines them with response-level gradients under the current policy.Source: §1 · pp. 1–2
  • Paired scoring of fixed response groups shows that shared normalization propagates score discrepancies across coefficients, while policy snapshots can yield different update directions and retained groups.These findings motivate characterizing coefficient effects on current-policy updates while preserving progress and avoiding directional deviation.Source: §1 · p. 2
  • SetGRPO combines joint coefficient-set calibration, geometry-guided safe recovery, and independent risk calibration to gate accepted updates while retaining the original GRPO objective.The global threshold controls conditional direction-reversal risk for accepted updates.Source: §1 · p. 2
  • SetGRPO achieves the best performance among compared training methods on both Qwen3.5 and Llama-3.2 model families across ten benchmarks and three domains.The evaluation reports consistent improvements across backbones and task domains.Source: §1 · p. 2

2. Related Work

Related work covers rubric-based feedback and GRPO, along with calibration and selective risk control. The paper distinguishes its focus on tracing grader-score discrepancies through GRPO updates and controlling reversal risk on accepted updates.

2. Related Work

  • Prior work examines reward over-optimization, judge consistency and bias, fine-grained evaluation, rubric-based training, and trajectory reuse, but does not trace score discrepancies through GRPO to actual policy updates.Related work also includes multimodal or embodied selection and evaluation and studies reasoning and user-engagement gaps.Source: §2 · p. 3
  • Prior studies also address reward-model capability, rubric feedback, and judge bias, while SetGRPO shifts calibration to joint training coefficients and induced policy updates.The section also notes related calibration and reliability work in retrieval and search settings.Source: §2 · p. 3
  • Conformal prediction and selective prediction provide precedents for finite-sample coverage and risk control on accepted subsets.The paper applies these ideas to GRPO training coefficients and the conditional direction-reversal risk of accepted updates.Source: §2 · p. 3

3. Preliminary

GRPO maps group rewards to response-level coefficients and combines them with current-policy gradients to form local updates. Paired pilot experiments show that score discrepancies couple coefficients, their update effects depend on policy geometry, and screening changes the accepted groups whose aggregate reversal risk matters.

3.1. Rubric-based GRPO

  • Group normalization maps grader rewards to training coefficients, which enter the original GRPO objective as response-level advantages.The mapping group-centers and scale-normalizes rewards, so each coefficient depends jointly on rewards across the group.Source: §3.1 · p. 4
  • The local update contribution of a coefficient vector depends on response-level gradients under the current policy, not coefficients alone.The preliminary analysis frames the propagation chain from score discrepancy through coefficients and gradients to group retention.Source: §3.1 · p. 4, §3.2 · p. 4

3.2. Preliminary Study on Grader-Induced Update Risk

  • Paired scoring of frozen response groups under the same grader and mapping both score sets through GRPO’s training transform provides the pilot study’s comparison.The study uses Qwen3.5-9B with the RubricHub training set and corresponding policy snapshots.Source: §3.2 · p. 4
  • A score change for one response affects the entire coefficient vector because GRPO uses shared group statistics, producing correlated rather than independent response-wise shifts.The study concludes that training uncertainty should be modeled for the joint coefficient vector.Source: §3.2 · pp. 4–5, §3.1 · p. 4
  • Similar coefficient shifts can produce different update magnitudes or directions under different policy snapshots, so coefficient-space distance alone does not determine update safety.Updates from the two scorings may even have a negative inner product.Source: §3.2 · p. 5
  • A per-group screening condition cannot by itself control aggregate direction-reversal risk because screening changes the distribution of groups entering optimization.Risk assessment therefore needs conditional statistics over the groups actually accepted.Source: §3.2 · p. 5

4. Methodology

SetGRPO models grader-induced shifts jointly in GRPO’s training coefficients, then uses the current policy’s update geometry and calibrated risk gating to recover safe updates. Its three modules calibrate coefficient uncertainty, find direction-preserving recovery ratios, and select which groups to update.

4.1. Joint Reward Set Calibration (JRSC)

  • Paired grader scores produce coefficient residuals for a joint uncertainty set, while a local reward-map Jacobian captures how perturbations affect coupled group coefficients.The set’s shape reflects the cross-response coupling induced by shared normalization.Source: §4.1 · p. 6
  • Independent paired residuals calibrate the ellipsoidal set’s radius, which under exchangeability covers second-scoring training coefficients with marginal probability at least 1 − α.The stated exchangeability condition requires identical data construction, response sampling, and scoring protocols with permutation-invariant joint distributions.Source: §4.1 · p. 6

4.2. Geometry-Guided Safe Recovery (GGSR)

  • GGSR maps the joint coefficient set into policy-update and reward spaces to find the largest recovery ratio whose update remains directionally consistent under the current policy.It anchors recovery at the feasible coefficient with the smallest local-update norm; if no ratio satisfies the conditions, the group produces no update.Source: §4.2 · pp. 6–7

4.3. Selective Risk-Gated Optimization (SRGO)

  • SRGO assesses direction-reversal risk conditional on each threshold’s accepted groups, then selects a risk-feasible threshold maximizing cumulative recovery.SetGRPO retains GRPO’s probability ratio, clipping, and token-level surrogate term, replacing only the training coefficients.Source: §4.3 · pp. 7–8

5. Experiments

Experiments compare SetGRPO with baseline and rubric-based training methods across benchmark domains and backbones. SetGRPO leads the reported Qwen3.5 macro averages, with further same-backbone comparisons showing gains on Qwen3 models.

5.1. Experimental Setup

  • Experiments train on RubricHub data and compare three backbones against Base and four training methods under shared training budgets and evaluation protocols.The ten downstream benchmarks span healthcare, writing, and instruction following; Macro Avg. aggregates their official metrics.Source: §5.1 · pp. 8–9

5.2. Main Results

  • SetGRPO achieves the highest Macro Avg. on Qwen3.5-4B (60.26) and Qwen3.5-9B (63.51), with gains across healthcare, writing, and instruction-following tasks.Relative to RubricHub, average gains on Qwen3.5-9B are 1.22, 2.61, and 0.96 points in those domains, respectively.Source: §5.2 · p. 9
  • During training, accepted-group ratios remain nonzero later, mean recovery ratios stay within 76%–80%, rewards rise, and policy entropy declines gradually without abrupt collapse.The accepted-group ratio is lower than in unfiltered training; screening excludes groups failing the projection condition while nonzero training coefficients still enter optimization.Source: §5.2 · p. 9

5.3. Same-Backbone Comparison

  • On Qwen3-4B, SetGRPO reaches a Macro Avg. of 57.56, 1.50 points above RubricHub; medical gains are larger on open-ended than multiple-choice evaluations.The HealthBench and LLMEval-Med gains are 2.48 and 2.45 points, while MedQA and MedMCQA differences are 0.47 and 0.46 points.Source: §5.3 · p. 10, Table 2 · p. 10
  • On Qwen3-4B, SetGRPO improves over RubricHub by 2.46 points on WritingBench and 1.71 on Creative Writing v3, and by 2.62, 2.41, and 2.37 on IFEval, IFBench, and MulDimIF.The InfoBench difference is 0.55 points.Source: §5.3 · pp. 10–11, Table 2 · p. 10
  • On Qwen3-14B, SetGRPO improves the Macro Avg. over RubricHub by 1.47 points.The paper reports consistent overall gains after calibrating the joint coefficient set and execution threshold separately for each backbone.Source: Table 2 · p. 10, §5.3 · p. 11

6. Analysis and Discussion

SetGRPO’s analysis evaluates risk calibration, computational cost, recovery behavior, affine invariance, empirical coverage, and screening under policy drift. It reports increased benchmark performance alongside additional computation, with accepted updates maintaining low reversal risk under the tested conditions.

6.2. Ablation Studies

  • The ablations compare joint versus isotropic coefficient sets, remove projection or recovery conditions, replace calibrated thresholding with a fixed threshold, and include matched random discard.The random-discard baseline retains the same number of groups as SetGRPO.Source: §6.2 · p. 11, Table 5 · p. 12

6.1. Risk Calibration and Retained Updates

  • At the selected threshold τsel = 0.25, the one-sided direction-reversal risk upper confidence bound remains below the target β = 0.08.As screening tightens, empirical risk and its upper confidence bound decrease overall.Source: Figure 4 · p. 11, §6.1 · p. 11

6.3. Efficiency Comparison

  • The ten-benchmark Macro Avg. rises from 55.41 to 60.26 (+4.75), while per-step time, peak VRAM, and cumulative GPU hours increase by 25.6%, 9.8%, and 25.4%.Rollout and grader times remain unchanged at 48 s and 18 s; the overhead mainly comes from Jacobian computation and per-group recovery search.Source: Table 6 · p. 12, §6.3 · p. 12

6.4. Recovery and Selection Diagnostics

  • Recovered coefficient norms remain generally stable without a sustained upward trend, while cumulative recovery continues across training stages.The rate of cumulative recovery varies with the number of groups meeting recovery conditions and their maximum recovery ratios.Source: §6.4 · pp. 12–13, Figure 5 · p. 13, Figure 7 · p. 13
  • Accepted updates meet both projection conditions: full-space projection exceeds τsel and reward-space projection is positive; rejected groups fail at least one condition.The joint screening retains updates consistent with reference training coefficients while maintaining positive reward-space projection.Source: §6.4 · p. 13, Figure 6 · p. 14

6.5. Affine-Invariance Sanity Check

  • Under four positive-affine reward transformations, coefficient cosine ranges from 0.9997 to 1.0000, update cosine from 0.9995 to 1.0000, with no decision flips.The largest recovery-ratio difference is 0.8×10−3; transformations that do not preserve within-group reward ordering may alter coefficients and screening quantities.Source: Table 7 · p. 14, §6.5 · pp. 14–15, Figure 8 · p. 15

6.6. Empirical Coverage and Risk Selection

  • At the final setting, nominal and empirical coverage are both 0.95; accepted groups have 0.047 empirical reversal risk and a 0.078 upper bound, within β.The finite-sample marginal coverage guarantee applies when calibration and covered groups satisfy exchangeability.Source: §6.6 · pp. 15–16, Figure 9 · p. 16

6.7. Checkpoint-wise Validation under Policy Drift

  • Across checkpoints, accepted groups retain positive recovery ratios and low reversal risk, while rejected groups consistently show higher reversal rates.Calibration quantities remain fixed while current-policy gradients and recovery geometry are recomputed at each checkpoint.Source: §6.7 · pp. 16–17, Figure 10 · p. 17

7. Representative Cases

Representative cases show that decisions track projection conditions and feasible recovery, rather than score discrepancies alone. Accepted, rejected, and recovery-infeasible examples illustrate how candidate updates are evaluated.

7. Representative Cases

  • A rejected HealthBench case passes the full-space condition (Mfull = 0.29) but fails the reward-space condition (Mreward = −0.06), yielding η★ = 0.00.Across the cases, score pairs do not vary monotonically with the decision; minimum projections and maximum feasible recovery ratio correspond directly to the screening conditions.Source: §7 · p. 17, Figure 12 · p. 20
  • SetGRPO combines coefficient uncertainty with current-policy response-level gradients to evaluate policy-update and reward-space projections and determine each group’s maximum recovery ratio.Groups meeting the projection conditions retain nonzero updates; groups failing them do not write back an update.Source: §7 · pp. 17–18

8. Conclusion

SetGRPO models grader-induced coefficient shifts jointly, recovers updates using current-policy geometry, and gates accepted updates by conditional reversal risk. Across ten open-ended benchmarks in three domains, it achieves the best macro average among compared methods on Qwen-3.5 and Llama-3.2.

8. Conclusion

  • SetGRPO uses a joint coefficient uncertainty set, geometry-guided recovery, and risk-gated selection to address grader stochasticity while preserving GRPO’s optimization structure.Across ten open-ended task benchmarks in three domains, it achieves the best macro-average performance among compared methods on Qwen-3.5 and Llama-3.2.Source: §8 · p. 22
  • Its finite-sample guarantee covers conditional direction-reversal risk for accepted updates under exchangeability, not privacy, fairness, factuality, or harmlessness.The authors also identify possible grader bias or drift, sensitive calibration data, and amplification of data or grader artifacts as risks.Source: §8 · p. 22

Related papers

Loading 2610.36807v1…