PaperlayerView exact revision on arXiv

Source-linked AI summary

Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rollouts

Esther Xin

arXiv:2609.06386v1cs.LGcs.CL

TL;DR

The paper examines whether verifier errors are independent across completions in a GRPO group, given that group-relative advantages rely on averaging verifier scores. It measures within-group dependence in real rollouts and finds substantial marginal correlation, while noting that prompt difficulty may also contribute.

  • Problem

    The group-relative advantage argument that averaging verifier scores attenuates individual errors requires independent errors, but dependence across completions answering the same prompt has not been established.

    Source: §1 · p. 1

  • Method

    The study measures verifier-error correlation across eight Qwen2.5 rollouts per prompt using certified answer-form categories, and replays advantages under rule-based verifier configurations.

    Source: §1 · pp. 1–2

  • Results

    0.530 pooled verifier-error correlation (95% CI [0.500, 0.560]) corresponds to a Kish effective sample size of 1.70 from eight rollouts.

    Source: §4.1 · p. 6

  • Takeaways & Limitations

    Marginal dependence relevant to realized group advantages is substantial and category-structured, supporting prompt- and answer-form-aware analysis of verifier noise.

    Source: §3.4 · p. 4

    Pooled correlation may reflect heterogeneous prompt difficulty as well as answer form; the analysis does not isolate residual correlation after conditioning on prompt difficulty and gold correctness.

    Source: §3.4 · p. 4

Abstract

from arXiv · show

Group-based reinforcement learning with verifiable rewards (RLVR) scoresmultiple completions per prompt using automatic verifiers. Analysesbased on independent verifier errors may overlook dependence associatedwith shared answer formats. We investigate this dependence in24,998 groups of eight completions generated by Qwen2.5-1.5B onMATH, GSM8K, and DeepMath-103K. We estimate a pooled within-groupverifier-error correlation of 0.530 (95% confidence interval:0.500--0.560). Under an exchangeable-error model, this correspondsto a design-effect-adjusted effective sample size of 1.70 for aneight-completion group. Dependence varies substantially across answerforms: fractions, radicals, symbolic expressions, and intervals exhibitstronger clustering than unit annotations and percent signs. Replayinggroup-relative advantages across four rule-based verifier configurationsidentifies at least one advantage-sign disagreement in up to 0.83% ofgroups. Because a group is repeated sampling for one prompt, thiswithin-group clustering may reflect shared prompt difficulty as well asshared answer form, and we do not attempt to separate the two here.Unlike studies of correlated judgments across multiple evaluators, ouranalysis examines dependence across completions scored by the sameverifier. These findings motivate prompt- and answer-form-aware analysesof verifier noise rather than characterizations based solely onaggregate error rates.

1 Introduction

The paper asks whether verifier errors are independent across completions in a GRPO group, a condition underlying the noise-robustness argument for group-relative advantages. It measures this dependence in real rollouts and examines its gradient-level and practical implications.

Contributions

  • The study measures within-group verifier-error correlation in real GRPO rollouts, extending certified answer-form error categories to training groups scored by one verifier.The question is whether category-concentrated errors persist across rollouts answering the same prompt and affect the policy gradient.Source: §1 · pp. 1–2

Relation to correlated error across judges

  • The analysis examines a different axis from correlated-judge studies: errors across one verifier’s scores of completions within a training group.Ensembling more verifiers does not address dependence that is a property of the training group.Source: §1 · p. 2

Contributions

  • The contributions include a preliminary comparison suggesting correlation increases with policy strength and a gradient-level degeneracy gap of roughly 6.3–6.5× over a homogeneous-difficulty i.i.d. reference.The gap is also consistent with between-prompt difficulty heterogeneity and is not attributed solely to correlated verifier error.Source: §1 · p. 2
  • A category-conditioned verifier router reaches 97.1% accuracy at 55.6% coverage on 5,556 independently human-labeled traces, alongside coverage-matched comparisons.The paper also releases a rollout corpus, verdict matrix, category labels, and analysis code.Source: §1 · p. 2

2 Related Work

Prior work studies verifier corruption, systematic error, and correlated errors across evaluator panels, but uses assumptions or settings distinct from dependence among real rollout completions scored by one verifier.

Verifier unreliability as a stochastic channel

  • Prior verifier-noise corrections model errors as i.i.d. or use population-level rates, whereas this work examines dependence within real rollout groups.The cited approaches include stochastic-channel corrections, Bernoulli-noise debiasing, and learning dynamics based on Youden’s index.Source: §2 · p. 2

Systematic verifier error

  • Prior studies show systematic verifier errors can affect training outcomes, while other work reports noisy-data robustness findings tied to verifier false negatives.The injected errors in one study are independent within experimenter-defined buckets, unlike emergent dependence in real rollout groups.Source: §2 · pp. 2–3

Rule- versus model-based verifiers; gradient starvation and biased advantage estimation

  • Prior work distinguishes rule-based false negatives from model-based verifier reward hacking; studies of group degeneracy and advantage bias use established baselines.The paper frames its degeneracy and bias measurements as gaps over those baselines rather than raw rates.Source: §2 · p. 3

Correlated error across evaluators

  • Correlated-error studies of judge panels and repeated benchmark trials motivate Kish effective-sample-size and ICC framing, but use different units than GRPO groups.Here the correlated units are rollouts scored by one verifier, not judges in a panel or repeated trials of one benchmark item.Source: §2 · p. 3

3 Method

The method samples grouped completions, labels answer forms, scores each completion under four rule-based verifiers, and measures group-level error dependence and advantage consequences. It also evaluates a category-conditioned verifier router using human-labeled traces.

3.1 Rollout corpus

  • The rollout corpus samples eight completions per prompt from Qwen2.5 models over a 50,000-prompt pool, retaining 24,998 complete 1.5B groups and a partial 7B-AWQ corpus.Groups with fewer than eight returned completions are dropped rather than backfilled.Source: §3.1 · p. 3

3.2 Category labels

  • Each extracted final answer receives one or more of 17 certified surface-form categories, with a highest-precedence primary category used for group-level analysis.Trailing periods, newlines, and internal whitespace are preserved as objects of study.Source: §3.2 · p. 3

3.3 Verifier configurations

  • Four rule-based configurations score completions: STRICT, LOOSE, NUMERIC, and FLEX, with FLEX covering answer forms NUMERIC marks inapplicable.FLEX adds percent-decimal equivalence, fraction parsing, and order-insensitive tuple, set, and list comparison.Source: §3.3 · p. 3

3.4 Within-group correlation statistic

  • The method estimates marginal intraclass correlation of verifier-error indicators and reports Kish design effects and effective sample sizes, with bootstrap intervals and category estimates limited to n ≥ 100 groups.Because pooled correlation includes prompts of varying difficulty, heterogeneous prompt error rates can induce positive marginal correlation even under conditional independence; prompt-level difficulty is not fully controlled.Source: §3.4 · pp. 3–4

3.5 Advantage-replay

  • Advantage replay compares verifier-specific group-relative advantages, degenerate-group rates against a homogeneous-difficulty i.i.d. reference, trace-level sign flips, and groups with at least one flip.The degeneracy comparison is descriptive, not causal; zero reward-derived advantage does not imply a zero total policy gradient because KL regularization remains.Source: §3.5 · p. 4

3.6 Category-conditioned router and human labels

  • The router selects the verifier with highest category-specific accuracy against human labels, using 5,556 stratified traces reviewed by two annotators.The category-selection rule and evaluation use the same labeled set without a held-out split.Source: §3.6 · p. 4

4 Results

Across 24,998 Qwen2.5-1.5B groups, verifier errors cluster within groups, especially for structurally or semantically complex answer forms. The section also reports exploratory policy-scale, group-degeneracy, and verifier-routing results with scope qualifications.

4.1 Verifier error is correlated within groups

  • 0.530 (95% CI [0.500, 0.560]) was the pooled within-group verifier-error correlation for Qwen2.5-1.5B, equivalent to an effective sample size of 1.70 from eight rollouts.The estimate was stable across collection sizes, reaching 0.530 at 24,998 groups.Source: Table 1 · p. 5, §4.1 · p. 6

4.2 Correlation concentrates in structurally complex categories

  • Formatting-only categories showed near-zero correlation, while structural or semantic categories showed correlations from 0.38 to 0.69.The contrast runs opposite to the answer forms that dominate aggregate false negatives in Xin (2026).Source: Figure 1 · p. 6, §4.2 · p. 6

4.3 Stability at small sample sizes

  • For large_numeric, the estimate moved from ρcat = 0.0 at 29 groups to 0.660 at 243 groups.This motivated the n ≥ 100 reporting threshold used in the paper and recommended for replication.Source: §4.3 · p. 6

4.4 An exploratory comparison across policy scale

  • ρcat was 0.657 for partial Qwen2.5-7B-AWQ versus 0.530 for Qwen2.5-1.5B, an exploratory comparison confounded by partial sampling and AWQ quantization.The category split also appeared in comparable 7B categories, but this was not independent confirmation.Source: Figure 2 · p. 6, §4.4 · p. 6

4.5 Group degeneracy exceeds a homogeneous-difficulty reference

  • Observed degeneracy was roughly 0.65 versus a 0.09–0.10 homogeneous-difficulty reference, a gap not attributed to correlated verifier error alone.Between-prompt difficulty heterogeneity can also inflate degeneracy; separating the contributions requires a difficulty-conditioned reference not computed here.Source: §4.5 · p. 7
  • Group corruption was at most 0.0083 for every pair of rule-verifier configurations.The four configurations mostly agreed on clean, boxed answers; larger disagreement appeared between rule verifiers and model-based judgment.Source: §4.5 · p. 7

4.6 A category-conditioned router, evaluated against human labels

  • No verifier was most accurate across every category: numeric-tolerance rules led on numeric and fraction forms, while CompassVerifier-3B led on symbolic and irrational forms.The router reached 97.1% accuracy at 55.6% coverage, but was not compared at matched coverage and its category rule was evaluated on its selection set.Source: Table 6 · p. 5, §4.6 · p. 7

5 Discussion

Within-group verifier-error dependence is substantial and category-structured, a quantity population-level error rates alone do not guarantee is accounted for. Because the dependence occurs across completions scored by one verifier, adding verifiers to an ensemble does not address it.

5 Discussion

  • The authors distinguish substantial marginal dependence within real training groups from prior analyses’ conditional independence assumption, whose internal validity they do not dispute.They identify marginal dependence as the quantity a GRPO-computed baseline sees.Source: §5 · pp. 7–8
  • Figure 4 plots router and single-verifier accuracy against coverage, but the comparison is not at matched coverage.The router’s accuracy therefore should not be read as a coverage-matched comparison.Source: Figure 4 · p. 8
  • Population-error-rate corrections alone do not guarantee that the substantial, category-structured marginal error dependence seen by a GRPO baseline is accounted for.The authors do not claim to disprove analyses that are valid conditional on independence.Source: §5 · p. 8
  • Adding verifiers to a majority vote does not address dependence across one verifier’s judgments within a training group; the group, not the panel, is the unit to decorrelate.The category-conditioned router selects one verifier rather than aggregating votes.Source: §5 · p. 8
  • The ∼6.5× gap to the homogeneous-difficulty reference exceeds pairwise sign-flip rates, plausibly because dependence makes whole groups uniformly right or wrong.This is a possible interpretation, not an established mechanism, because the analysis does not separate correlated verifier error from between-prompt difficulty heterogeneity.Source: §5 · p. 8

6 Limitations

The reported scope is constrained by an exploratory, confounded 7B comparison, taxonomy and stability-threshold choices, and evaluation caveats; the claim concerns measured error structure, not demonstrated training outcomes.

6 Limitations

  • The 7B comparison is exploratory and cannot establish a general increase in verifier-error correlation with policy strength.It uses a partial corpus, differs in quantization from the 1.5B run, and has no bootstrap confidence interval for the pooled estimate.Source: §6 · p. 8
  • The answer-form taxonomy was validated on synthetic certified rewrites rather than natural rollouts, and the n ≥100 stability threshold was selected after instability appeared.The authors note identical extraction and categorisation code was used on both corpora.Source: §6 · p. 8
  • The paper does not establish a training outcome because it does not run training to convergence.Its claim is limited to measured verifier-error structure and its gradient consequence.Source: §6 · p. 8
  • Perfect agreement across 200 overlapping annotations does not rule out shared annotator bias.The authors recommend larger-scale randomized, blinded double annotation with preregistered adjudication for future router evaluation.Source: §6 · pp. 8–9
  • The router’s category-selection rule was fit and evaluated on the same labeled set, so its reported accuracy is in-sample rather than evidence of generalization.The ρcat, Kish effective-sample-size, and advantage-replay results do not depend on those labels.Source: §6 · p. 9

7 Conclusion

The conclusion reports substantial within-group verifier-error correlation structured by answer form, while qualifying the degeneracy comparison and router evaluation. It calls for closer scrutiny of the independence assumption in RLVR reliability work.

7 Conclusion

  • ρcat = 0.530 (95% CI [0.500, 0.560]) measures substantial within-group verifier-error correlation structured by independently established answer-form categories.The categories are certified and established independently of rollout data (Xin, 2026).Source: §7 · p. 9
  • Group-level degeneracy is roughly 6.5× the homogeneous-difficulty i.i.d. reference, but ordinary between-prompt difficulty heterogeneity could also explain the gap.The gap should not be attributed to correlated verifier error alone.Source: §7 · p. 9
  • A category-conditioned router reaches accuracy close to the best single verifier while covering substantially more traces, at modestly higher query cost than that verifier’s coverage.The category-selection rule was evaluated in-sample, and a held-out replication would strengthen the result.Source: §7 · p. 9
  • The findings do not invalidate existing noise-correction methods but support closer scrutiny of their independence assumption and of training groups in future RLVR reliability work.Source: §7 · p. 9
Loading 2609.06386v1…