A Step Harder, Not a Step Too Far: The Benefits and Limits of Curriculum Difficulty Progression in GRPO

1 Introduction

• Sample difficulty changes with the evolving policy, so initial estimates can become misaligned and miss substantial within-band turnover.Relatively stable global rankings can coexist with substantial turnover within specific bands.Source: §1 · pp. 2, 4
• Because medium-solvability samples are more likely to yield mixed correct and incorrect responses, the introduction asks whether persistent selection is enough or harder bands help.Such mixed responses provide informative group-relative signals.Source: §1 · p. 2
• The experiments track 1,495 GSM8K samples, test a static curriculum, and compare online fixed-medium selection with staged progression across difficulty bands.The fixed-medium setting keeps B3 active, while the staged curriculum reselects samples as the active band shifts.Source: §1 · pp. 2–3, Figure 2 · p. 4
• Progressing from fixed-medium selection to a moderately harder band improves test performance, but further progression to harder samples reduces it.The paper distinguishes informative medium-difficulty learning signals from strongest final performance.Source: §1 · pp. 3–4

2.1 GRPO and RLVR

• GRPO derives relative advantages from multiple responses to one prompt, while prior work has focused more on objectives and training stability than individual-sample difficulty dynamics.GRPO reduces reliance on explicit value estimation compared with PPO-based approaches.Source: §2.1 · p. 5

2.2 Dynamic Data Selection

• Online selection methods adapt sample choices using model-dependent signals, but whether persistent intermediate-solvability selection maximizes final performance remains unclear.The paper addresses this gap by tracking the same samples as the policy evolves.Source: §2.2 · pp. 5–6

2.3 Curriculum Learning with Model-Dependent Difficulty

• Prior curricula organize tasks by estimated or model-dependent difficulty, but GRPO's relative-reward signals leave open whether harder bands keep helping as difficulty rises.Very easy or very difficult problems can both yield limited informative comparisons.Source: §2.3 · p. 6

3.1 Difficulty Under the Current Policy

• Sample solvability is defined under the current policy, with empirical success rate used to estimate solvability and corresponding difficulty.Table 1 summarizes notation for policy-dependent difficulty and curriculum settings.Source: §3.1 · pp. 6–7, Table 1 · p. 7

3.2 Difficulty Dynamics

• Difficulty dynamics are tracked across an ordered checkpoint sequence using success-rate span and maximum deviation from the initial estimate.Table 2 summarizes the notation for these dynamics and band-membership metrics.Source: §3.2 · pp. 7–8, Table 2 · p. 8
• Direction reversals are counted after changes smaller than δ are discarded; δ = 0.125 is the primary threshold, with alternative thresholds and band-membership measures also examined.The primary threshold corresponds to four increments of 1/32 in empirical success rate.Source: §3.2 · pp. 8–9

3.3 Evaluation of Static Difficulty and Curriculum Scheduling

• The static curriculum ranks samples by initial solvability and keeps that ordering fixed to test whether initial difficulty estimates remain aligned with the evolving policy.Higher initial solvability indicates an easier sample.Source: §3.3 · p. 9
• A difficulty band is a predefined range of policy-dependent sample difficulty, and the active band determines which samples contribute to the policy loss.The framework asks whether online re-estimation should accompany fixed-medium selection or progression to harder bands.Source: §3.3 · p. 9
• The fixed-medium curriculum keeps B3 while reselecting eligible samples online, whereas the staged curriculum moves toward lower-success-rate, harder bands.The gating mechanism defines experimental conditions; an effective update requires at least one sample to pass the gate and a parameter update to occur.Source: §3.3 · p. 9

4.1 Model and Training Configuration

• GSM8K experiments use Qwen2.5-1.5B-Instruct with LoRA; difficulty estimation uses 32 responses per problem, while curriculum optimization uses 8.Correctness is the primary verifiable reward, with an auxiliary length reward of fixed weight 0.2; the test set is reserved for generalization evaluation.Source: §4.1 · pp. 9–10, Table 3 · p. 10

4.2 Difficulty Tracking and Static Curriculum

• All 7,473 training problems receive 32 initial generations, and a reference GRPO run saves checkpoints from step 0 through 7,473.A fixed, proportionally sampled observation set is reevaluated at checkpoints to track policy-dependent solvability.Source: §4.2 · p. 10
• The static curriculum fixes a descending initial-solvability ranking, then reevaluates the next 500 scheduled samples at six later training steps.Static and standard GRPO comparisons report best observed test accuracy retrospectively, without using test values to guide training.Source: §4.2 · pp. 10–11

4.3 Fixed-Medium and Staged Curricula

• The staged curriculum moves through B1 to B5 using mutually exclusive diagnostic subsets of 128 problems per band, excluded from training.Stage transitions use a patience rule: diagnostic accuracy must improve by more than 0.0107, followed by two additional evaluations.Source: §4.3 · p. 11
• The fixed-medium condition keeps B3 = [0.375, 0.625] active while reselecting samples online under the evolving policy.Its best observed test accuracy is used only for post-hoc comparison, not to guide training or parameter updates.Source: §4.3 · pp. 11–12
• Three shared random seeds are used, and staged performance is compared at the selected B4 checkpoint and after 100 additional effective updates in B5.Multi-seed results are reported as mean ± standard deviation.Source: §4.3 · p. 12

5.1 State Dependence and Non-Monotonicity of Difficulty

• Mean empirical success rate rose from 0.6554 to 0.7975, while 53.38% of samples reversed direction at δ = 0.125.The mean training-period span was 0.2770, and 54.72% of samples had a span of at least 0.25.Source: §5.1 · p. 12, Figure 3 · p. 13

5.2 Global Ordering Persistence and Local Band-Membership Change

• Final rank correlation remained 0.8092, but only 20.74% of initial B3 members remained there, while 65.00% of final B3 members entered from outside.The initial and final B3 sets had Jaccard similarity 0.1497.Source: §5.2 · pp. 13–14, Figure 4 · p. 14

5.3 Precomputed Difficulty Misalignment in Static Curriculum Training

• At step 5500, planned mean success rate was 0.3986 versus a current mean of 0.5573, while the static curriculum achieved 0.6522±0.0022 test accuracy.The authors interpret the mismatch as temporal misalignment, not evidence that it alone causes performance degradation.Source: §5.3 · pp. 14–15

5.4 Performance Across Difficulty Bands

• In the detailed run, fixed-medium reached 0.6459, B4 reached 0.6558, and B5 declined to 0.6353 after 100 additional effective updates.B5 diagnostic accuracy also fell from 0.5215 at entry to 0.4512 after 100 additional updates.Source: Table 5 · p. 15, §5.4 · p. 15

5.5 Robustness Across Random Seeds

• Across three seeds, B4 reached 0.6591 ± 0.0077 versus fixed-medium 0.6449±0.0024; B5 fell to 0.6405 ± 0.0053.B4 also exceeded standard GRPO and static-curriculum performance in each seed.Source: Table 6 · p. 16, §5.5 · p. 16

6 Discussion

• Sample difficulty changes substantially and non-monotonically with the policy, so initial estimates may become misaligned even when global rankings remain relatively stable.The competitive performance of the static curriculum shows that misalignment alone does not make a precomputed curriculum ineffective.Source: §6 · p. 17
• Persistent medium-difficulty selection does not achieve the strongest observed performance: moderate progression helps, whereas further progression to harder samples consistently reduces performance.The pattern suggests a non-monotonic relationship between curriculum difficulty and model performance.Source: §6 · p. 17
• Online sample selection and difficulty progression are distinct components: one identifies samples in a current-policy band, while the other chooses the active band over training.The results support combining online re-estimation with staged progression, while controlling how far difficulty increases.Source: §6 · p. 18
• The evidence is limited to one model family and GSM8K; online solvability uses eight responses, and staged-curriculum comparisons evaluate complete procedures rather than isolated factors.The staged condition also uses fixed diagnostic subsets, a predefined transition criterion, and a smaller candidate pool.Source: §6 · p. 18

7 Conclusion

• Sample difficulty changes with the evolving policy, supporting current-policy difficulty estimates for online sample selection.The conclusion also reports that initial-policy estimates may become misaligned with the current policy.Source: §7 · p. 18
• Across three random seeds, progressing to a moderately harder band provides additional gains, whereas further progression to a harder band consistently reduces performance.The conclusion characterizes the benefit of increasing curriculum difficulty as non-monotonic.Source: §7 · p. 18

Appendix A Detailed Difficulty-Dynamics Metrics

• The metrics characterize local difficulty-band changes and non-monotonic trajectories for bands defined by associated success-rate intervals.The appendix introduces these as complementary measures of band-membership change.Source: Appendix A · p. 19
• Retention, outflow, inflow, Jaccard similarity, and mismatch characterize persistence, departure, entry, overlap, and disagreement in band membership.Mismatch normalizes the symmetric difference between initial and current band membership by the fixed observation-set size N.Source: Appendix A · p. 20
• The appendix distinguishes monotonic solvability movement from extra variation caused by back-and-forth changes.Ei is zero for a monotonic trajectory and positive when variation exceeds the observed solvability span.Source: Appendix A · p. 20
• Re-entry counts a sample that leaves band B and later returns; its denominator includes only samples observed in B that subsequently left it.The appendix defines re-entry among samples that have left the band at least once, not among all samples or all observations.Source: Appendix A · p. 20

Appendix B Full Difficulty-Dynamics Statistics

• Table B1 reports the magnitude of sample difficulty changes over the complete GRPO trajectory.The supplied caption identifies the table's scope but does not provide its values.Source: Table B1 · p. 21
• Table B2 examines how the direction-reversal rate responds to the minimum-change threshold.The caption does not state the resulting sensitivity or values.Source: Table B2 · p. 21
• Table B3 reports rank consistency between initial and current empirical success rates.The caption identifies the comparison but does not report its outcome.Source: Table B3 · p. 21
• Table B4 reports membership changes in the medium-difficulty band B3 = [0.375, 0.625].The caption identifies B3 as the medium-difficulty band.Source: Table B4 · p. 21
• Table B5 reports re-entry rates across the five difficulty bands.The caption does not provide the rates or compare bands.Source: Table B5 · p. 21

• The initial and final solvability ranks for 1,495 tracked GSM8K problems show a Spearman rank correlation of ρ = 0.8092.The concentration near the dashed y = x line is consistent with this correlation.Source: Figure B1 · p. 22
• The axes show solvability-rank percentiles at the initial and final checkpoints, with 0 indicating harder problems and 100 indicating easier problems.Source: Figure B1 · p. 22
• Color denotes the number of problems in each bin.Source: Figure B1 · p. 22

Diagnostic setup

• Each band has a fixed 128-problem diagnostic subset, evaluated every 50 effective updates with eight responses per problem; these subsets are excluded from training.The subsets are constructed sequentially from B1 to B5, excluding problems assigned to earlier subsets.Source: Appendix C · p. 22

Stage selection

• A later checkpoint becomes the new best only if diagnostic accuracy exceeds the current best by more than 0.0107; after that, two evaluations 50 and 100 updates later determine a plateau.The retained best checkpoint initializes the next band; if a new best occurs during the patience window, the 100-update criterion restarts.Source: Appendix C · pp. 22–23

Diagnostic trajectories

• No post-entry B5 checkpoint is selected for any seed, and diagnostic accuracy declines after B5 entry in all three runs.Seed 1 falls from 0.5215 to 0.4971 and 0.4512; Seed 2 from 0.5127 to 0.5086 and 0.4521; Seed 3 from 0.5745 to 0.5713 and 0.5273, at entry, 50, and 100 effective updates, respectively.Source: Appendix C · p. 23, Figure C1 · p. 24

Difficulty misalignment

• Figure D1 compares planned and current-policy mean solvability and plots their mean absolute deviation at evaluated training steps.Source: Figure D1 · p. 25

Static curriculum

• The static curriculum's initial difficulty estimates are compared with current-policy estimates at each evaluation point, yet it remains a competitive baseline across seeds.Table D11 reports its best observed test performance across random seeds.Source: Appendix D · pp. 25–26, Table D10 · p. 25

Staged trajectory

• Table D12 presents Seed 1's staged-curriculum test trajectory across difficulty bands B3, B4, and B5.Source: Appendix D · p. 26, Table D12 · p. 26

Fixed-medium trajectory

• For fixed-medium runs, the active band remains B3 = [0.375, 0.625] while samples are reselected online; test accuracy does not guide training or sample selection.The reported best values are retrospective maxima along completed trajectories.Source: Appendix D · p. 26
• Fixed-medium test accuracy peaked at 0.6459 (Seed 1, 450 effective updates), 0.6422 (Seed 2, 700), and 0.6467 (Seed 3, 650).These are the best observed values along the completed trajectories.Source: Appendix D · p. 26, Table D13 · p. 27