PaperlayerView exact revision on arXiv

Source-linked AI summary

When Does Closed-Loop Curriculum Adaptation Help GRPO? An Analysis on Countdown

Yuv Raj Pant, Moshiur Rahman Tonmoy, Kaiqun Fu

arXiv:2610.39056v1cs.LGcs.AI

TL;DR

GRPO can receive weak learning signals from problems that are too hard or too easy, motivating curriculum adaptation to current capability. CLOSEDLOOPGAUSSIAN adjusts a Gaussian easy-to-hard sampler’s pace and spread using periodic per-difficulty probes. On Countdown, it reports higher mean exact-match accuracy than fixed E2H-G and vanilla GRPO across four seeds, with the largest reported gains at intermediate difficulties.

  • Problem

    GRPO can yield weak advantage signals when problems are too hard for mostly failed rollouts or too easy for uniformly correct responses.

    Source: §I · p. 1

  • Method

    CLOSEDLOOPGAUSSIAN uses per-difficulty pass rates to adapt the pace and spread of a Gaussian easy-to-hard sampler for GRPO.

    Source: §I · p. 2

  • Results

    At n = 3, mean accuracy was 52.3% for CLOSEDLOOPGAUSSIAN versus 40.8% for Fixed E2H-G; overall mean exact-match accuracy exceeded Fixed E2H-G and vanilla GRPO across four seeds.

    Source: B. Accuracy Across Difficulty Levels · p. 4, §V · p. 5

  • Takeaways & Limitations

    The study frames training-time data selection as an adaptive compute-allocation decision while leaving the reward function, model architecture, and GRPO optimizer unchanged.

    Source: §V · p. 5

    The evaluation covers one 3B model, Countdown, and four seeds; controller thresholds, gains, and bounds are manually calibrated.

    Source: §IV · p. 5

Abstract

from arXiv · show

Reinforcement fine-tuning has become an effective post-training paradigm for improving the reasoning ability of large language models, but it remains sample-inefficient when the training distribution is mismatched to the model's current capability. In group-relative objectives such as Group Relative Policy Optimization (GRPO), problems that are too hard often produce uniformly failed rollouts, while problems that are too easy provide little useful advantage signal. In this work, we introduce CLOSEDLOOPGAUSSIAN, a lightweight adaptive curriculum method for efficient reinforcement fine-tuning on ordered reasoning tasks. Instead of following a fixed easy-to-hard schedule, CLOSEDLOOPGAUSSIAN preserves the structure of an Easy-to-Hard Gaussian sampler while dynamically adjusting its pace and spread using periodic probe estimates of per-difficulty accuracy. The controller accelerates training when the model is ahead of a reference curriculum, slows down when the model falls behind, and broadens sampling when earlier difficulties regress. This feedback mechanism keeps training focused on tasks that are challenging but still learnable, without changing the reward function, model architecture, or underlying GRPO optimizer. Experiments on symbolic arithmetic reasoning task show that CLOSEDLOOPGAUSSIAN improves exact-match accuracy over vanilla GRPO and fixed open-loop curricula, with the strongest gains in intermediate-difficulty regimes where group-relative learning signals are most informative. Codes are available at https://github.com/yuvrajpant56/Closed_Loop_Curriculum.

I. INTRODUCTION

GRPO training is sensitive to task difficulty: very hard or very easy problems yield weak learning signals. CLOSEDLOOPGAUSSIAN adapts an easy-to-hard curriculum using model feedback, and its four-seed Countdown evaluation reports improved average accuracy under an identical rollout budget.

I. INTRODUCTION

  • GRPO estimates advantages from multiple responses to the same prompt, avoiding a separately trained critic [4].RLVR scores sampled solutions with verifiers such as exact-answer matching or executable checks [1] [2] [3].Source: §I · p. 1
  • Very hard problems often yield mostly failed rollouts, while very easy problems yield uniformly correct responses, weakening GRPO’s within-group learning signal.Expected policy improvement is reported to be highest at intermediate difficulty [7].Source: §I · p. 1
  • Methods that alter optimization or rollout procedures address learning-signal extraction, while online filtering can spend variable rollout cost per update.The cited examples include DAPO [6], GSPO [8], and ShareGRPO [9].Source: §I · pp. 1–2
  • Fixed easy-to-hard schedules do not adjust their pace or spread to the model’s actual learning trajectory, which may vary across seeds and difficulty levels.Prior adaptive schedulers are described as using a task-selection policy or a single target-difficulty variable rather than separately controlling advance and rehearsal breadth.Source: §I · p. 2
  • CLOSEDLOOPGAUSSIAN adjusts the pace and spread of a Gaussian easy-to-hard sampler using per-difficulty pass rates.The controller adapts both how quickly difficulty advances and how broadly earlier difficulties are rehearsed.Source: §I · p. 2
  • The paper evaluates CLOSEDLOOPGAUSSIAN against vanilla GRPO and a fixed open-loop curriculum on Countdown across four seeds under an identical rollout budget.The reported comparison shows improved average accuracy, especially where group-relative learning signals are most informative.Source: §I · p. 2

II. METHODOLOGY

CLOSEDLOOPGAUSSIAN changes the prompt-sampling distribution for GRPO-style reasoning post-training while leaving the model, verifier, reward function, and optimizer unchanged. Periodic probe estimates guide adjustments to curriculum pace and spread, reallocating a fixed rollout budget across difficulty levels.

II. METHODOLOGY

  • CLOSEDLOOPGAUSSIAN changes only the prompt-sampling distribution, leaving the model, verifier, reward function, and GRPO optimizer unchanged.Periodic probes estimate accuracy by difficulty bucket, and a feedback controller adjusts curriculum pace and spread.Source: §II · p. 2
  • The sampler, trainer, and probe evaluator share one scheduler, so controller updates affect the next sampled batch and reallocate rollout compute by difficulty.Sampling can advance toward harder problems, slow when performance lags, or broaden rehearsal when earlier difficulties regress.Source: §II · p. 2

A. Ordered Difficulty Space

Countdown examples are assigned to ordered difficulty buckets by operand count, and the curriculum samples from those fixed groups. A Gaussian sampler distributes probability across buckets, with pace and spread adapted over time.

A. Ordered Difficulty Space

  • Countdown instances use each input number exactly once in an arithmetic expression that evaluates to the target integer.The allowed operations are addition, subtraction, multiplication, and division.Source: A. Ordered Difficulty Space · p. 2
  • Difficulty groups are fixed and ordered by operand count n ∈ {2, 3, 4, 5}, with larger n empirically harder for the policy.The curriculum samples a bucket and then an example uniformly from that bucket; it does not infer or relabel difficulty.Source: A. Ordered Difficulty Space · p. 2

B. Gaussian Curriculum Sampler

  • The Gaussian easy-to-hard sampler retains the fixed E2H-G sampling family, making feedback the primary design difference between the methods.At each optimizer step, the sampler assigns probabilities across the ordered difficulty buckets.Source: B. Gaussian Curriculum Sampler · p. 2
  • Pace β_t controls how quickly the curriculum center advances, while spread σ_t controls rehearsal breadth across neighboring buckets.A positive probability floor keeps every bucket available, while the hardest bucket’s share remains negligible early in training.Source: B. Gaussian Curriculum Sampler · p. 2

C. Closed-Loop Curriculum Controller

The controller uses periodic probe accuracies and fixed feedback rules to adjust the pace and spread of a Gaussian curriculum around a reference schedule. These updates respond to competence and regressions while keeping the schedule bounded.

Probe evaluation and calibration

  • Every 100 optimizer steps, greedy probes on 100 held-out examples per difficulty bucket estimate bucket-wise accuracy.The probe uses the unchanged Countdown verifier and is the only additional GPU computation introduced by the controller.Source: C. Closed-Loop Curriculum Controller · p. 3

Pace control

  • Bucket-specific thresholds classify performance as ahead of or behind the reference pace, and weighted signals adjust progression accordingly.The pace target is clipped, EMA-smoothed, and anchored to the reference; positive feedback accelerates progression, while negative feedback slows it.Source: C. Closed-Loop Curriculum Controller · p. 3

Spread control

  • Spread control tracks each bucket’s best probe accuracy and broadens sampling when designated easier buckets regress.Because the spread signal is nonnegative, spread control broadens sampling on regression and otherwise relaxes toward its reference.Source: C. Closed-Loop Curriculum Controller · p. 3

Training procedure

  • At each probe interval, the training loop updates pace and spread for subsequent sampling, while keeping both bounded around their reference values.Between probes, the current pace and spread remain unchanged.Source: C. Closed-Loop Curriculum Controller · p. 3

A. Experimental Setup

Experiments use LoRA-fine-tuned Llama-3.2-3B-Instruct with GRPO for 1,600 steps and periodic greedy probes. The controller adds inference time in the reported representative run.

Model and optimization

  • Experiments fine-tune meta-llama/Llama-3.2-3B-Instruct with LoRA rank 32, scale 64, and dropout 0.05 for 1,600 optimizer steps.Training uses a 10−5 learning rate, cosine decay, and a 0.03 warm-up ratio; GRPO samples four generations per prompt with KL coefficient 0.001.Source: A. Experimental Setup · p. 3

Generation and probing

  • Greedy controller probes run every 100 steps on 100 examples per difficulty bucket; representative run times were 12.94 h for CLOSEDLOOPGAUSSIAN, 11.49 h for Fixed E2H-G, and 10.82 h for GRPO.The passage describes these single-run timings as indicative only.Source: A. Experimental Setup · p. 3

Controller configuration

  • The controller uses reference values βref = 0.50 and σref = 0.50, with β bounded to [0.35, 0.70] and σ to [0.40, 0.80].Forgetting is monitored over buckets D0 and D1.Source: A. Experimental Setup · p. 3

B. Accuracy Across Difficulty Levels

Across four seeds, CLOSEDLOOPGAUSSIAN has higher mean exact-match accuracy than Fixed E2H-G and GRPO. Its largest mean gains over Fixed E2H-G occur at intermediate difficulty levels, although the four-seed comparison is descriptive.

B. Accuracy Across Difficulty Levels

  • Across four seeds, CLOSEDLOOPGAUSSIAN achieved 42.4% ± 14.0% average exact-match accuracy, versus 35.0% ± 11.4% for Fixed.Source: B. Accuracy Across Difficulty Levels · p. 3, Table I · p. 4
  • At n = 3, CLOSEDLOOPGAUSSIAN reaches 52.3% versus 40.8% for Fixed E2H-G; at n = 4, it reaches 23.2% versus 16.3%.CLOSEDLOOPGAUSSIAN also exceeds Fixed E2H-G at n = 2 (82.7% vs. 73.1%) and n = 5 (11.5% vs. 9.7%).Source: B. Accuracy Across Difficulty Levels · p. 4
  • CLOSEDLOOPGAUSSIAN exceeds Fixed E2H-G on all four seeds, but large dispersion and only four seeds support descriptive, not statistical-significance, interpretation.Each difficulty bucket has 250 evaluation problems, so one example corresponds to 0.4 percentage points.Source: B. Accuracy Across Difficulty Levels · p. 4

C. Controller Dynamics: When Does Adaptation Help?

Probe trajectories show different adaptation responses across seeds: some accelerate toward harder buckets, while another broadens rehearsal after easier-bucket forgetting. A single-seed sampler analysis also shows a shift in exposure toward the hardest bucket.

C. Controller Dynamics: When Does Adaptation Help?

  • On seeds 42, 43, and 1, rising n = 3 probe accuracy accompanies positive pace signals and βt falling toward approximately 0.35–0.40, while σt stays near reference with modest widening.Seed 43 has the strongest acceleration and highest final accuracy; seed 1 has a transient mid-training dip before returning to acceleration.Source: C. Controller Dynamics: When Does… · p. 4
  • On seed 44, persistently low n = 3 competence returns the pace signal to zero after step 200 and βt near reference, while easier-bucket forgetting widens σt to approximately 0.63.This is the reference-anchoring behavior when no pace-adjustment signal is present.Source: C. Controller Dynamics: When Does… · p. 4
  • In seed 43 sampler logs, CLOSEDLOOPGAUSSIAN assigns 6.45 percentage points more sampling mass to n = 5 than Fixed E2H-G, mainly from n = 3 and n = 4.n = 3 accuracy improves despite less n = 3 training exposure in this single-seed analysis.Source: C. Controller Dynamics: When Does… · p. 4

D. Diagnostic Sensitivity to Controller Authority

The controller diagnostic finds that locking spread can coincide with lower accuracy on the selected seed, while seed-dependent results show curriculum arms avoiding GRPO’s near-floor outcomes. These findings are diagnostic rather than a tight estimate of expected gain.

D. Diagnostic Sensitivity to Controller Authority

  • Seeds 42, 43, and 1 move ahead of pace and accelerate the curriculum, while seed 44 widens the distribution to counter forgetting.The figure tracks pace, spread, the pace control signal, and n=3 probe accuracy across four seeds.Source: Figure 2 · p. 4
  • On selected seed 43, locking σ=0.50 reduced average accuracy from 58.9 to 26.2, while the default [0.35, 0.70] pace band outperformed tighter and wider alternatives.This post-hoc diagnostic omits the complementary β-locked/σ-active condition and does not establish an average contribution of spread or a causal decomposition of pace and spread.Source: D. Diagnostic Sensitivity to… · p. 4

E. Seed-Dependent Stability

  • Vanilla GRPO fell to 7.7% on seeds 44 and 1, while Fixed E2H-G stayed above 19.2% and CLOSEDLOOPGAUSSIAN above 25.1%.The reported account attributes the large standard deviations to GRPO training instability and cautions that four seeds on one model and task provide an ordering and mechanistic account, not a tight estimate of expected gain.Source: E. Seed-Dependent Stability · pp. 4–5

IV. LIMITATIONS AND FUTURE DIRECTIONS

The study reports higher mean exact-match accuracy than fixed E2H-G and vanilla GRPO across four seeds, but its evidence is limited in scope. It identifies broader evaluation and controller calibration as future work.

IV. LIMITATIONS AND FUTURE DIRECTIONS

  • Evidence is limited to one 3B model, Countdown, and four seeds, so broader model, task, and seed coverage is needed to refine expected-gain estimates.Direct comparisons with adaptive curricula are also left for future work.Source: §IV · p. 5
  • Controller thresholds, gains, and bounds are manually calibrated; overly strict settings can cause curriculum deadlock, motivating automated calibration.The paper says its reference-anchored bounded design mitigates this failure mode.Source: §IV · p. 5

V. CONCLUSION

  • CLOSEDLOOPGAUSSIAN achieved higher mean exact-match accuracy than fixed E2H-G and vanilla GRPO across four reported seeds, with substantial seed-to-seed variation.The conclusion describes the scheduler as reallocating a fixed rollout budget across difficulty buckets without changing the reward function, model architecture, or GRPO optimizer.Source: §V · p. 5

Related papers

Loading 2610.39056v1…