Beyond Uncertainty: Multi-Solver Disagreement Rewards for Self-Evolving Reasoning Curricula

1 Introduction

• The Challenger–Solver loop generates problems near the Solver’s capability frontier and improves the Solver through increasingly difficult questions, avoiding the expense of large-scale human-curated training data.R-Zero rewards questions according to Solver sampling uncertainty and reports gains of +6.49 on math and +7.54 on general reasoning from Qwen3-4B-Base.Source: §1 · p. 1
• Single-solver uncertainty can rate a question as easy when it aligns with that model’s learned biases, obscuring genuine difficulty for differently trained models.R-Zero’s pseudo-label accuracy declined from 79% to 63% across three iterations, with performance degrading after iteration 3 [7].Source: §1 · p. 1
• The proposed normalized Shannon entropy over a heterogeneous ensemble rewards questions whose solvers produce conflicting plurality answers.It is a drop-in reward replacement requiring no training-framework modifications.Source: §1 · p. 1
• +1.34 points average on MATH-500, AMC, and Olympiad accompany a −0.60-point GSM8K regression attributed to reduced curriculum size.The reward signal trends upward from 0.61 to 0.72 within one training iteration, while uncertainty remains near 0.20.Source: §1 · p. 1

2 Related Work

• R-Zero [7] uses solver uncertainty in a data-free Challenger–Solver loop, while Absolute Zero Reasoner [22] uses single-model self-play and Socratic-Zero [18] uses three-agent co-evolution.STaR [20] bootstraps from self-generated rationales, and ReST [4] uses quality-filtered self-generated data.Source: §2 · p. 2
• Prior ensemble methods also guide neural-network sample selection and pseudo-label selection, while curriculum and self-paced learning order data by difficulty or model confidence [5].Source: §2 · p. 2
• Disagreement here steers a generative RL curriculum toward harder questions, unlike prior uses that select or filter samples from a fixed pool.Query-by-committee selects samples where committee members disagree; ensemble diversity has theoretical foundations in Dietterich.Source: §2 · p. 2
• The paper situates its reasoning evaluation among RL elicitation [2], GRPO optimization [15], process reward models [9], and benchmarks including MATH and GSM8K [1].It uses Qwen3 [19] and Llama [10] as foundation models.Source: §2 · p. 2

3 Notations and Preliminaries

• Qθ denotes the Challenger parameterized by θ, and E denotes an ensemble of N solver variants.For each generated question, each solver variant produces k sampled answers.Source: §3 · p. 2
• The composite Challenger reward combines uncertainty and disagreement rewards with a repetition penalty: r(q) = α·runc(q)+β·rdis(q)−rrep(q).The plurality answer for each variant is computed by majority vote with symbolic-equivalence matching.Source: §3 · p. 2
• Challenger and Solver training alternate, with each model frozen while the other is trained, so question difficulty and solver capability advance together.In R-Zero, the alternating updates use GRPO [15].Source: §3 · pp. 2–3

4.1 Uncertainty Reward (Baseline)

• The baseline uncertainty reward peaks when the pooled Solver answers are split and becomes zero when answers are consistent, even if the question is genuinely difficult for another model.The paper identifies its limit as dependence on the variance of a single model’s sampling distribution.Source: §4.1 · p. 3

4.2 Ensemble Disagreement Reward

• The disagreement reward uses normalized entropy over plurality answers from a three-variant ensemble spanning sampling temperatures and model architectures.Each variant generates 10 answers; the largest symbolic-equivalence cluster determines its plurality answer. Unanimity yields 0, three distinct answers yield 1, and a 2-vs-1 split yields approximately 0.58.Source: §4.2 · p. 3

4.3 Composite Reward

• The composite Challenger reward combines uncertainty computed over the pooled bag of all 3k=30 samples with cross-variant disagreement, using α=1.0 and β=0.5; uncertainty remains the primary signal.Malformed outputs without valid boxed content receive r(q)=−1 as a format penalty.Source: §4.3 · p. 3

4.4 Complementarity of 𝑟unc and 𝑟dis

• Uncertainty captures collective sampling noise, whereas disagreement captures different conclusions across variants that can each answer consistently.The paper illustrates a 2-vs-1 split where the pooled uncertainty reward is 0.33 while disagreement is approximately 0.58.Source: §4.4 · p. 3

4.5 System Design

• The reward system dispatches questions concurrently to three inference engines, adding approximately 8 minutes per batch and roughly 3× total GPU-hour overhead versus single-solver evaluation.Timeouts degrade computation to responding variants only; parse failures receive a fixed penalty, and repetition clustering uses successfully parsed questions.Source: §4.5 · p. 3
• +1.34 points average improvement on competition math is reported for the disagreement curriculum.Table 1 reports solver accuracy (pass@1 %), with all models evaluated using the same pipeline and mathruler answer grading.Source: Table 1 · p. 4

5.1 Experimental Setup

• The protocol uses Qwen3-4B-Base, a standard R-Zero iter-1 baseline, and one additional Challenger iteration before Solver training and evaluation.The ensemble combines two Solver v1 variants at temperatures 0.7 and 1.0 with Llama-3.2-3B-Instruct at temperature 0.7.Source: §5.1 · p. 4

5.2 Results

• +1.34 points average on MATH-500, AMC, and Olympiad; GSM8K declines by 0.60 points.The reported deltas compare the disagreement-Challenger curriculum with R-Zero under the same evaluation pipeline.Source: §5.2 · p. 4

5.3 Reward Dynamics

• Within the single Challenger iteration, r_dis rises from 0.609 to 0.718 (+18%) as its standard deviation falls from 0.336 to 0.278 (17%).The reward has minor non-monotonic fluctuations in early batches.Source: §5.3 · p. 4, Table 2 · p. 4
• By contrast, r_unc remains approximately 0.20 (range 0.191–0.209), while r_rep averages 0.001–0.004.The authors interpret the low repetition penalty as indicating diversity without a significant BLEU-penalty contribution.Source: §5.3 · p. 4
• Whether disagreement reward avoids collapse across multiple iterations remains untested; the reported dynamics cover one iteration of eight batches.Source: §5.3 · p. 4

• The upward disagreement-reward trajectory and +1.34 points on MATH-500, AMC, and Olympiad suggest inter-model disagreement captures a learnable difficulty dimension complementary to single-solver uncertainty.The reward uses normalized Shannon entropy over a heterogeneous solver ensemble and requires no changes to the underlying RL framework.Source: §6 · p. 4

• Claude Sonnet 4.6 served as an auxiliary answer-recheck judge alongside mathruler symbolic equivalence checking; generative AI also supported proofreading and language editing.The authors state that generative AI was not used for ideation, experimental design, or data generation.Source: GenAI Usage Disclosure · p. 4