PaperlayerView exact revision on arXiv

Source-linked AI summary

Wrong Votes, Right Gradients: Why Majority-Vote Rewards Track Verifiable RL

Dylan P. Wilson, Chengsong Huang, Jiaxin Huang

arXiv:2610.20898v1cs.LGcs.AIcs.CL

TL;DR

Self-generated rewards can recover much of the gain from verifiable rewards, yet majority vote returns a possibly wrong modal answer. The paper analyzes shared-parameter batch updates and finds that incoherent majority-vote errors can yield gold-aligned updates, while the equivalence remains local.

  • Problem

    Self-generated rewards can perform well, but prompt-level accounts cannot explain how majority-vote training helps when the modal answer is wrong.

    Source: §1 · pp. 1–2

  • Method

    The paper decomposes majority-vote updates into shrinkage and misdirection relative to gold updates, measuring gradient agreement and error coherence across prompts.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    Across five models, majority-vote updates agree with gold as closely as independent gold updates do, with noise-corrected cosine 1 within error.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Self-generated labels can stand in for verification when residual errors are prompt-specific and incoherent, but shared errors are a boundary to that equivalence.

    Source: §7 · p. 10

    The equivalence is local: gold training ends ahead on held-out majority accuracy, and the short, unregularized runs do not establish trajectory equivalence.

    Source: §1 · p. 2, §7 · p. 10

Abstract

from arXiv · show

Reinforcement learning with self-generated labels, such as majority vote over a model's own samples, often recovers most of the gain of training with verifiable rewards. This is puzzling from the usual prompt-level view, in which majority vote only returns a model's modal answer and cannot add information about correctness. We argue that the relevant unit is the batch-level update. Because parameters are shared, a prompt's own gradient accounts for less than 1\% of the change in that prompt's output at practical batch sizes, so a wrong label matters only to the extent that label errors share a direction across prompts. We decompose the majority-vote update into shrinkage and misdirection relative to the gold update, and show that misdirection vanishes with batch size whenever errors are incoherent across prompts, however often the labels are wrong. Measuring per-response gradients on five models, we find that majority-vote errors are as incoherent as a random-reward null, that the majority-vote update agrees with the gold update as closely as two independent gold updates agree with each other, and that its noise-corrected cosine with the gold update is 1 within error. Randomly flipping up to 30\% of gold labels leaves the update direction intact, while the same fraction flipped by a shared rule destroys it, and rewards that favor a kind of response on every prompt, such as a learned reward model, fall measurably short. The equivalence holds at every checkpoint we examine but is local: gold-reward training still ends ahead on held-out majority accuracy. Error coherence, not error rate, determines when self-generated labels can stand in for verification.

1 INTRODUCTION

Self-generated rewards can approach verifiable-reward gains, despite majority vote returning a model’s possibly incorrect modal answer. The paper argues that shared-parameter batch updates—not prompt-level correctness alone—explain when this works.

1 INTRODUCTION

  • Self-generated rewards often recover most gains from ground-truth rewards, but can also collapse or reinforce preferences unrelated to correctness.Verification, by contrast, is constrained to labeled data or domains with an automatic oracle.Source: §1 · p. 1
  • Majority vote returns a model’s modal answer, which may be wrong, so a prompt-level account cannot explain improvements on problems where that answer is incorrect.The passage reports a held-out majority becoming correct about three times as often as resampling alone.Source: §1 · pp. 1–2
  • At practical batch sizes, a prompt’s own gradient accounts for less than 1% of its output change, making cross-prompt error alignment—not error rate—the determinant of update misdirection.Incoherent errors average out as batch size grows, whereas errors sharing a direction leave a floor.Source: §1 · p. 2
  • Across five models, majority-vote updates match gold-update agreement as closely as independent gold updates, with randomly flipped labels tolerated but shared response preferences falling short.The noise-corrected cosine is 1 within error; randomly flipping up to 30% of gold labels leaves direction unchanged.Source: §1 · p. 2
  • The equivalence is local: gold-reward training still ends ahead on held-out majority accuracy, and shared residual errors can limit when self-generated labels stand in for verification.The proposed criterion is most favorable when errors are sparse and specific to individual prompts.Source: §1 · p. 2

2 PRELIMINARIES

The paper compares majority-vote and gold rewards through shared-parameter GRPO updates computed on the same sampled responses. It benchmarks their agreement against the variability between independent gold updates.

2 PRELIMINARIES

  • GRPO samples prompts and response groups, scores responses, and forms group-relative advantages that determine a summed batch update.At the sampling policy, importance ratios equal one and clipping is inactive.Source: §2 · p. 2
  • Gold rewards compare each final answer with a reference, while majority-vote rewards compare it with the group’s most frequent answer.The two rewards coincide when the majority answer is correct; their differences occur on prompts with a wrong majority label.Source: §2 · pp. 2–3
  • The main training comparison uses Qwen3-4B-Base on DeepScaleR with 256 prompts and 16 responses per step, alongside gradient analyses on four additional models.Both training runs share prompts and initialization and omit KL and entropy terms.Source: §2 · p. 3
  • The analysis stores per-response log-probability gradients as 216-dimensional count sketches, allowing different rewards to reweight identical samples.Sketch cosines agree with exact fp32 gradients within 0.004.Source: §2 · p. 3
  • Gold-update agreement across disjoint prompt halves provides a ceiling because prompt-specific gradients keep independent gold updates from being parallel.The halves are paired to avoid sharing sampling noise between majority-vote and gold updates.Source: §2 · p. 3
  • The noise-corrected estimate c∞ targets the cosine between expected majority-vote and gold updates, and is reported only when its denominator is sufficiently stable.The stated reporting threshold is Z ≥ 0.30.Source: §2 · p. 3

3 GRADIENT-LEVEL COMPARISON OF MAJORITY VOTE AND GOLD

Across models, majority-vote updates align with gold updates at least as closely as independent gold updates do, and their expected direction matches gold. Batch-level effects explain why wrong labels can have little influence on individual outputs.

3 GRADIENT-LEVEL COMPARISON OF MAJORITY VOTE AND GOLD

  • At initialization, replacing gold labels with majority vote changes the batch update no more than prompt sampling already does.The comparison covers all five models.Source: §3 · p. 3
  • Majority-vote updates reach the independent-gold agreement ceiling within 0.02 on four of six arms, with the largest raw gap on Llama-3.1-8B-Instruct.That arm’s cross-half cosine is 0.399 against a 0.438 gold ceiling; noise correction gives c∞ = 1.013.Source: §3 · p. 3, Table 1 · p. 4
  • As prompt groups B grow, majority-vote cosine approaches c∞, estimated at 1.00 to 1.01 within 95% intervals across five models.The Figure 1 curves show cosine to the summed gold gradient against B.Source: Figure 1 · p. 4, §3 · p. 4
  • At B = 64 to 128, a prompt’s own gradient accounts for 0.09–0.38% of its output change on Qwen3 models, and 0.83% on Llama-3.2-3B-Instruct at B = 64.On Qwen3-1.7B-Base, a small step along its own gradient changes a prompt’s output only 1.7 to 1.9 times as much as another prompt’s gradient.Source: §3 · pp. 4–5
  • On wrong-majority prompts in the first Qwen3-4B-Base batch, the MV update increases expected gold reward on 82% to first order, as other prompts outweigh the mislabeled term.The other prompts contribute +0.075 to the cosine with the prompt’s gold gradient, nearly the gold-label contribution of +0.074.Source: §3 · p. 5

4 WHY LABEL ERRORS CANCEL

The section explains why majority-vote (MV) errors can cancel across prompts: update direction depends on error coherence, not error rate. It tests this account against random rewards and alternative reward structures.

4.1 SHRINKAGE AND MISDIRECTION

  • The update decomposes into shrinkage, which rescales the gold step, and orthogonal misdirection, which changes its direction.Incoherent errors make misdirection vanish for large batches regardless of error rate; coherent errors leave a floor.Source: §4.1 · p. 5
  • Independent flips of gold labels preserve the expected update direction for any flip probability below one half, adding zero-mean noise across prompts.The disattenuated cosine remains 1 under this condition.Source: §4.1 · p. 6

4.2 MV ERRORS ARE INCOHERENT

  • MV error alignment is 30–60 times lower than gold-gradient alignment and is comparable to the Gaussian random-reward null.Batch-sum growth independently gives the error an exponent of 1/2, while the gold exponent rises to 0.87 at B ≈150.Source: §4.2 · p. 6, Table 2 · p. 6

4.3 ERROR STRUCTURE, NOT ERROR RATE

  • At a matched 20% error rate, random flips yield c∞=1.011, but shared-rule flips yield 0.287; at 31%, shared-rule flips yield −0.129.Rewards favoring response types across prompts also fall below 1, while pure length reward yields c∞=−0.415.Source: §4.3 · pp. 6–7

5 TRAINING DYNAMICS

The paper finds that MV errors remain incoherent during training and updates remain close to gold updates, yet local equivalence does not make the training trajectories identical. Gold training finishes ahead on held-out majority accuracy.

5 TRAINING DYNAMICS

  • MV errors remain at the random-reward null through steps 10 and 20, while their growth exponent stays at 1/2 and the gold exponent reaches 0.90 at step 20.The error-to-signal ratio and cosine at step 20 are reported as 0.24 and 1.000 [0.972, 1.030], respectively.Source: §5 · p. 7
  • At step 20, MV delivers 98% of the gold component, with a 14° rotation; on identical prompts, misdirection halves and batch cosine rises from 0.898 to 0.972.The error-to-signal ratio falls from 0.43 at initialization to 0.24 at step 20.Source: §5 · p. 7, Table 4 · p. 9
  • Through step 15, training accuracy is indistinguishable between runs, but the MV-reward-minus-accuracy gap grows to 0.15 in the MV run versus 0.09–0.10 in the gold run.The comparison uses a prompt-paired difference of +0.002 [−0.001, +0.006] over 3,840 pairs.Source: §5 · p. 7
  • MV repairs 13 of 66 initially wrong-majority level-5 MATH-500 problems by step 20, a 19.7% repair rate about three times the independent-half resampling rate.The evaluation problems are nearly disjoint from training, and removing the three with notable overlap leaves the repair rate within its interval.Source: §5 · p. 7

4.4 DO THE ERRORS STAY INCOHERENT ACROSS STEPS?

  • Across 36 pairs of training steps, MV-error cosine is 0.030, versus 0.001 for random rewards and 0.400 for gold gradients; error accumulation has exponent 0.594 versus gold’s 0.762.The authors note that these are first-order comparisons across gradients evaluated at different parameters, and that the horizon is short.Source: §4.4 · p. 8
  • At step 10 on level-5 problems, gold reaches maj@64 0.590 versus MV’s 0.541; on all MATH-500 at step 20, gold reaches 0.845 versus MV’s 0.772.Doubling the MV batch to 512 prompts does not close the gap.Source: §4.4 · p. 8

6 RELATED WORK

Related work studies self-generated rewards and label-free reinforcement learning, whose gains and failure modes remain unsettled. This paper distinguishes its batch-level explanation from prior prompt-level accounts.

Self-generated rewards

  • Self-generated rewards include self-consistency fine-tuning, self-judging, majority vote, confidence or entropy signals, and self-play task generation.Examples include Huang et al. (2023), Yuan et al. (2024), and Zuo et al. (2026).Source: §6 · p. 9

Label-free reinforcement learning

  • Label-free training can collapse, reinforce model preferences, lose correct answers under wrong consensus, or gain from spurious rewards or contamination.Prior explanations are prompt-level (Zhang et al., 2025a).Source: §6 · p. 9
  • Rather than propose a new reward, the paper asks whether reward-source errors share a component across prompts, framing the explanation at batch level.The framing connects label-noise robustness (Rolnick et al., 2017) with shared gradient directions (Chatterjee, 2020; McCandlish et al., 2018).Source: §6 · p. 9

7 CONCLUSION AND LIMITATIONS

The conclusion treats the batch-level update as the right unit: prompt-specific errors cancel, while shared error structure is reinforced. This criterion is local, and the paper’s gradient-level findings do not establish equivalent training trajectories.

7 CONCLUSION AND LIMITATIONS

  • A wrong majority matters when errors across prompts share a direction; measured residual errors are sparse, prompt-specific, and cancel in aggregate.Structure shared across prompts is reinforced.Source: §7 · pp. 9–10
  • Verification should matter most for recurring errors, such as widely shared misconceptions or errors accumulated by revisiting the same prompts.For label-free methods, the authors suggest current-policy labeling and fresh, diverse prompts to make errors less coherent.Source: §7 · p. 10
  • The runs use one model, are short and unregularized, and show sharply rising MV entropy beyond step 15, a regime the paper does not analyze.Only gradient-level results cover five models from two families.Source: §7 · p. 10
  • Per-step equivalence does not establish equivalent trajectories; over nine measured steps, errors accumulated slightly faster than a random walk.The authors also note two runs per reward and raw-gradient cosines, which may differ from directions reweighted by Adam’s preconditioner.Source: §7 · p. 10

A EXPERIMENTAL DETAILS

The appendix documents the training and evaluation setup, gradient measurements, and checks on the analyses. It also reports sensitivity to label-flip structure, checkpoint behavior, and possible evaluation–training overlap.

A EXPERIMENTAL DETAILS

  • The runs share a checkpoint, data order, and 256-prompt steps split into two 128-prompt optimizer updates; both use std-normalized advantages.Table 5 lists the configuration shared by gold and MV runs.Source: A EXPERIMENTAL DETAILS · p. 15, Table 5 · p. 15
  • Responses are scored from the last boxed answer in the final 400 characters; unparseable responses receive zero and are excluded from voting, while ties go to the first answer.Evaluation uses 64 samples per problem on MATH-500 and its 134 level-5 problems; maj@k is the accuracy of the most frequent parsable answer.Source: A EXPERIMENTAL DETAILS · p. 15
  • On identical 256 step-21 prompts scored on separate on-policy rollouts, step 20 leaves shrinkage unchanged, halves misdirection, and reproduces measured batch cosine; 95% bootstrap bands overlap at every batch size.Source: Table 6 · p. 16, Figure 3 · p. 17
  • Mean-centered and standard-deviation-normalized advantages give the same c∞ answer; step 10 is omitted because its split-half agreement is about 0.25.The standard-deviation weighting is the one used in the training loss.Source: Table 7 · p. 16
  • Independent flips preserve c∞ at 1 across models, while shared-rule flips lower it monotonically and turn Qwen3-1.7B-Base away from gold at ε = 0.30.The figure plots noise-corrected c∞ because the schemes differ in split-half agreement; raw cosines can rank them misleadingly.Source: Figure 2 · p. 16
  • Parsing improves through the first eight gold-run steps and then declines slowly, while majority-answer accuracy stays flat and votes become more concentrated.The independently seeded MV run parses at 0.87 at step 10 and 0.86 at step 12.Source: A EXPERIMENTAL DETAILS · p. 17, Table 9 · p. 18
  • The overlap check found one exact and 22 near matches among 500 MATH-500 problems; excluding three flagged level-5 problems changed the repair rate by less than its bootstrap interval.Gradient sketches use 216 dimensions and agree with exact fp32 gradients within 0.0039; including gold-silent groups raises MV norm and misdirection without changing its disattenuated cosine.Source: A EXPERIMENTAL DETAILS · p. 18
  • Split-half estimates average X, Y, and Z over 200 prompt-group partitions, with intervals from 250 bootstrap resamples and 12 partitions per resample.Table 8 compares two MV and two gold runs at step 10 on the same level-5 problems, with maj@k computed from random subsets for k < 64.Source: Table 8 · p. 17, A EXPERIMENTAL DETAILS · p. 18

B PROOFS

The proofs analyze how summed gradient components scale with batch size and show that independently flipped labels preserve the gold update direction in expectation. They also define a split-half correction for comparing expected updates despite prompt-sampling noise.

B PROOFS

  • Proposition 1 derives the squared norm of summed vectors from diagonal and off-diagonal terms, then differentiates with respect to log B to obtain α(B).The expression separates individual-vector scale from mean pairwise alignment.Source: B PROOFS · p. 18
  • For Proposition 2, wrong-majority prompts alone contribute to the orthogonal error component, which is analyzed against the summed gold gradients as batch size grows.The wrong-majority set contains εB prompts; the supplied expression is incomplete, so no limiting value is reported.Source: B PROOFS · pp. 18–19
  • Independent gold-label flips make the expected update parallel to the gold update, yielding a limiting cosine of 1.The result assumes independent flips across prompts; the expected update is scaled by (1 − 2ε).Source: B PROOFS · p. 19
  • The disattenuated cosine compares MV and gold expected updates using independent half-batch samples, with cross-half inner products isolating their alignment from sampling noise.Noise inflates the norm denominators, and the norm factors cancel in the ratio described in the passage.Source: B PROOFS · p. 19

C ADDITIONAL RESULTS

Additional results examine label freshness, update agreement, gradient coherence, and benchmark recovery. They show that on-policy MV labels retain more of the gold update component than stale labels, while the reported comparisons also mark scope and measurement qualifications.

C ADDITIONAL RESULTS

  • At step 20, frozen-base labels deliver 63% of the gold component (β = 0.37), versus 98% for on-policy MV labels, despite similar label accuracy.Frozen-base labels also add more misdirection: R⊥ = 0.29 versus 0.24.Source: C ADDITIONAL RESULTS · p. 19
  • Table 10 summarizes recovery of ground-truth RL gains by label-free methods, using benchmark-averaged scores and including only online MV variants.The caption defines recovery as (method − base)/(GT-RL − base); it provides no cell values here.Source: Table 10 · p. 19
  • TTRL’s recovery comparison overstates recovery relative to other rows because it trains on evaluation questions while Oat-Zero trains on MATH-train.Source: C ADDITIONAL RESULTS · p. 19
  • MV and gold updates cannot be distinguished by the split-half comparison when the 95% interval for their agreement difference contains zero.Table 12 reports the full split-half results behind Table 1; its caption defines the interval as (MV vs. gold) minus (gold vs. gold).Source: Table 12 · p. 20
  • Table 14 tracks majority-answer changes on 134 level-5 MATH-500 problems, comparing step 20 with the base model and a resampling null.The caption specifies 66 initially wrong-majority and 68 right-majority problems, with majorities based on 64 samples.Source: Table 14 · p. 20
  • RESTRAIN exceeds GT-RL on MMLU-STEM and GPQA-Diamond for Qwen3-4B, while its recovery on the four math benchmarks alone is about 80%.Source: C ADDITIONAL RESULTS · p. 21
  • Table 17 compares cross-step MV-error coherence with a random-reward floor and reports an exponent for error growth.The caption specifies that coherence is mean cosine over ten step pairs and that the random reward’s expected update is zero.Source: Table 17 · p. 22
  • The own-gradient share of a prompt’s output change is reported as falling only slightly from batch size 8 to 16 for Llama-3.2-3B-Instruct.The cited passage attributes the small decline to noisy small-batch gradients; Tables 18 and 19 define related own-share measures and scaling quantities.Source: C ADDITIONAL RESULTS · p. 22, Table 19 · p. 22, Table 18 · p. 22

Related papers

Loading 2610.20898v1…