PaperlayerView exact revision on arXiv

Source-linked AI summary

Cool the Sampler, Not the Learner: Sampling Temperature Moves the Staleness Cliff of Importance-Corrected GRPO

Taiheng Pan

arXiv:2609.36953v1cs.LGcs.CL

TL;DR

Importance-corrected GRPO can degrade before a stale sampler is refreshed, raising the question of how long correction remains usable. The paper cools only the sampler, keeping the learner’s objective unchanged. Cooling extends usable refresh intervals and can preserve short-interval learning, though its benefit has limits.

  • Problem

    The paper asks how long a sampler can remain unrefreshed while truncated importance correction repairs its mismatch with the learner.

    Source: §1 · p. 1

  • Method

    Decoupled cooling samples at Tb=0.8 while learner, reference model, and importance weights stay at temperature 1, recording the tempered behaviour probability.

    Source: §1 · p. 2, §2 · p. 3

  • Results

    At 1.5B, cooled 192-step refreshes matched uncooled 96-step refreshes at 0.857 final accuracy; at 7B, cooled 144-step runs ended at 0.92 and 0.93.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Sampling temperature and refresh interval should be chosen together, with cooling’s benefit depending on task and training stage.

    Source: §7 · p. 9

    At three times the safe interval, no tested temperature stayed stable through 300 steps, and cooling did not triple the safe interval.

    Source: §4 · p. 7

Abstract

from arXiv · show

Production RL for language models lets the sampler fall behind the learner and repairs the resulting mismatch with a truncated importance weight. We ask how long the sampler can go without a refresh under that correction, and find a cliff: on Qwen2.5-Math-1.5B and GSM8K, importance-corrected GRPO refreshed every 192 updates learns well for 180 steps and then degrades severely in all three data seeds before the refresh arrives. Published remedies for staleness act on the update; we act on the sampler instead. Decoupled cooling draws samples at temperature 0.8 while the learner, the reference model and the importance weights stay at temperature 1, with the behaviour probability recorded from the tempered distribution, so the learner's objective is unchanged. All corresponding cooled runs are stable, and the longer interval keeps what the short one delivered: at the same update budget, a cooled sampler refreshed every 192 steps matches an uncooled sampler refreshed every 96 at the end of training (0.857 for both) and averaged over it (0.79), whereas lowering the learning rate to a safe value ends 3-7 points lower. On Qwen2.5-Math-7B the degradation points at interval 192 predict that an interval of 144 is fatal without cooling and survivable with it; on two data seeds the uncooled runs degrade before their first refresh and the cooled runs pass it and end at 92-93% against 68-81%, with one cooled run degrading transiently late in the second cycle. The benefit has a window: at three times the safe interval and in a high-mismatch MATH setting cooling delays degradation without preventing it, stronger cooling is not better, and cooling without the correction collapses. Sampling temperature is a control on staleness tolerance, and temperature and refresh interval should be chosen together.

1 INTRODUCTION

Importance-corrected GRPO can fail before a refresh arrives, even after a long period of healthy learning. Decoupled cooling extends the usable interval while preserving short-interval learning, but its benefit depends on the setting.

1 INTRODUCTION

  • On GSM8K, all three 1.5B seeds reached about 0.81 before falling at steps 185–190, ahead of the 192-step refresh, and ended at 0.61, 0.59, and 0.47.Validation remained healthy through the first 150 steps; the paper calls this failure before cycle-end the staleness cliff.Source: §1 · p. 1
  • Decoupled cooling samples at Tb=0.8 while the learner, reference model, KL penalty, and importance weights remain at temperature 1, with the tempered behaviour probability recorded.This leaves the learner’s objective untouched.Source: §1 · p. 2
  • At 300 updates, a cooled sampler refreshed every 192 steps matched an uncooled sampler refreshed every 96 at 0.857 final accuracy and averaged 0.790 versus 0.787.Lowering the learning rate to a safe value ended 3–7 points lower in the same budget.Source: §1 · p. 2
  • At 7B, the predicted 144-step interval was fatal without cooling but survivable with it: cooled runs ended at 0.92 and 0.93 versus 0.68 and 0.81 uncooled.One cooled run degraded transiently late in its second cycle, so passing a refresh was not a permanent guarantee.Source: §1 · p. 2
  • Cooling moves the cliff but does not remove it: at three times the safe interval every tested temperature degrades, and high-mismatch settings can still collapse.Stronger cooling is not consistently better, and cooling without importance correction only postpones collapse.Source: §1 · pp. 2–3
  • The paper frames decoupled cooling as a sampler-side intervention that preserves the learner’s objective while moving the staleness cliff across two model sizes.It also reports limits and comparisons with update-side stabilisers.Source: §1 · p. 3

2 SETUP

The setup emulates a lagging sampler by holding its weights fixed between refreshes, applies token-level truncated importance correction, and separates sampler temperature from the learner’s objective.

2 SETUP

  • Runs use GRPO with 16 prompts and 8 samples per prompt, group-normalised advantages from rule-based correctness rewards, and one optimiser step per full batch.The experiments use Qwen2.5-Math-1.5B and 7B on GSM8K and MATH, with greedy validation.Source: §2 · p. 3
  • Sampler weights are refreshed every N ∈ {96, 144, 192, 288} learner steps; between refreshes, the sampler is frozen and staleness is t mod N.Runs last 300 steps, covering two or three refreshes.Source: §2 · p. 3
  • TIS applies min(πθ(at | st)/q(at | st), 2) as a fixed per-token policy-gradient coefficient, correcting gradients on sampled responses rather than which responses are drawn.The truncation caps the mismatch the weight can undo.Source: §2 · p. 3
  • In decoupled sampling, qTb is proportional to the sampler distribution raised to 1/Tb, and the recorded behaviour probability is qTb while learner and reference evaluations stay at temperature 1.Cooling changes proposed samples and estimator bias and variance, but keeps the target objective fixed.Source: §2 · p. 3
  • Degradation onset is the first validation point at or below the running maximum minus 0.10 after step 30; severe degradation uses a 0.15 drop.A drop is transient if the curve returns within 0.05 of its prior maximum within 20 steps, and sustained otherwise.Source: §2 · pp. 3–4

3 THE STALENESS CLIFF, AND MOVING IT

Importance-corrected GRPO develops a staleness cliff at longer refresh intervals, while decoupled cooling lowers learner–sampler mismatch and can move the cliff. The 1.5B and 7B results show both the benefit and its boundaries.

3.1 1.5B: NO CLIFF AT 96, A CLIFF AT 192, NO CLIFF AT 192 WHEN COOLED

  • At 192 steps on 1.5B, all three uncooled seeds severely degraded before refresh, while cooled seeds ended at 0.848, 0.857, and 0.859.At 96 steps, the three uncooled seeds ended at 0.861, 0.860, and 0.864; cooled runs at 192 matched their end accuracy and whole-run mean (0.857 versus 0.857; 0.790 versus 0.787).Source: §3.1 · p. 4

3.2 THE MISMATCH, ACROSS EVERY RUN

  • Within a cycle, learner–sampler KL grows roughly exponentially, resets at refresh, and grows faster with hotter sampling; KL crossing one nat preceded or accompanied every fixed-temperature corrected-run collapse described here.Early KL growth extrapolated from steps 20–60 did not distinguish runs that later degraded from those that did not.Source: §3.1 · p. 4, §3.2 · p. 4
  • Cooling lowered fixed-step KL by 1.7–7.8 times and lengthened time to 0.1 nat from about 100 to 150–175 steps on 1.5B and about 20 to 40 on 7B.Figure 2 compares fixed-step KL by model, task, and interval, then marks threshold-crossing time against the first refresh.Source: §3.2 · p. 4, Figure 2 · p. 5
  • Cooling was not a disguised learning-rate reduction: cooled runs had 2–8 times smaller learner KL to the initial policy at step 90 while validation accuracy was equal or higher.The reported 1.5B KL values were 0.004–0.007 versus 0.010–0.015 nats; at 7B they were 0.005–0.011 versus 0.029–0.043.Source: §3.2 · p. 4

3.3 7B: THE CLIFF ARRIVES EARLIER, AND A PREDICTION ABOUT MOVING IT

  • At 192 steps on 7B, cooling moved severe degradation from step 120 to step 180 and made collapse recoverable, but did not prevent first-cycle degradation.The observed degradation times motivated the 144-step prediction tested on two seeds.Source: §3.3 · pp. 4–5
  • At a 144-step refresh interval, the 7B prediction held on two seeds: cooled runs ended at 0.92 and 0.93 versus 0.68 and 0.81 for uncooled runs.The cooled run on one seed oscillated late in the second cycle, so cooling did not make that cycle uniformly safe.Source: §3.3 · p. 5

4 THREE OBJECTIONS, AND WHERE THE CLIFF CANNOT BE MOVED

Cooling compares favorably with tested update-side alternatives, but its benefit has limits: it delays degradation at longer staleness and under high mismatch without always preventing it. The figures and diagnostics show that the useful temperature range depends on the task and setting.

Update-side alternatives

  • At refresh interval 192, both µ-GRPO seeds are stable, but cooled TIS leads it by 3.5 and 0.6 points and does not add to it.µ-GRPO reduces mismatch and veto rate without raising accuracy under this recipe and budget.Source: §4 · pp. 5–6, Figure 4 · p. 6

7B test

  • On Qwen2.5-Math-7B, cooled runs at refresh interval 144 avoid pre-refresh degradation and end at 0.92–0.93, while uncooled runs degrade before refresh.At interval 192, the uncooled run collapses to zero and the cooled run degrades later, then recovers after refresh.Source: Figure 3 · p. 6

Update-side alternatives

  • Cooling at learning rate 2×10−6 ends 3.4 points above the best tested lower learning rate and 6.8 above the lower one within a 300-update budget.The same-budget comparison favors cooling for the stability–progress trade-off; Table 1 reports the comparison.Source: §4 · p. 5, Table 1 · p. 7

Temperature boundary

  • At refresh interval 288, degradation onset is not monotone in temperature, and in high-mismatch MATH the 0.6 setting is worse than 0.8.On standard MATH at refresh interval 192, no run degrades and cooling is neutral.Source: §4 · p. 6, Figure 5 · p. 7

Larger staleness

  • At refresh interval 288 on 1.5B GSM8K, no tested temperature stays stable through 300 steps; cooling to 0.8 moves degradation from step 190 to 250.Colder settings do not move onset further, no temperature triples the safe interval, and single runs vary by tens of steps.Source: §4 · p. 7, Figure 5 · p. 7

A harder task

  • On high-mismatch MATH, uncooled runs collapse in both seeds, while cooling delays degradation but does not prevent it; at standard settings neither temperature degrades.With doubled learning rate and refresh interval 288, cooled runs end at 0.41 and 0.39; at standard learning rate and interval 192, cooling is neutral.Source: §4 · p. 7, Figure 5 · p. 7

Cooling without the correction

  • At refresh interval 96 on GSM8K, cooled plain GRPO without importance correction collapses in all three seeds, while cooled TIS and uncooled TIS are stable.On MATH, the uncorrected cooled run dips at step 200 and recovers by step 240.Source: Figure 6 · p. 8, §4 · p. 8

Diagnostics

  • KL excursions, entropy collapse or explosion, and gradient spikes accompany degradation, but a one-nat KL crossing is not a reliable alarm.The crossing leads degradation by 9–42 steps in uncooled TIS runs, follows it by 16–27 steps in Tb=0.7 runs, and is absent in the collapsing controller run.Source: §4 · p. 8

5 RELATED WORK

The paper situates decoupled cooling among methods that address stale-sampler mismatch by changing the update, while its approach changes the proposal and leaves the objective untouched. It also contrasts stale sampling with on-policy temperature control and asynchronous staleness bounds.

5 RELATED WORK

  • Yao et al. (2025) introduced the truncated token-level importance weight used here, while Song et al. (2026) relate collapse time to accumulated drift.Other cited approaches constrain weight moments, scale steps by effective sample size, use relaxed clipping or vetoes, or adjust the proximal target or gradient direction.Source: §5 · p. 8
  • Unlike prior remedies that act on the update, decoupled cooling acts on the proposal and leaves the objective untouched.The paper compares this approach with µ-GRPO and tests their combination in Section 4.Source: §5 · p. 8
  • On-policy temperature findings concern a setting where sampler and learner share one policy, whereas this paper studies stale sampling and identifies different temperature costs.The cited on-policy work reports Tb=0.6 destabilising early and Tb=1.2 training more slowly but more stably; other work varies or learns temperature.Source: §5 · p. 9
  • Asynchronous trainers bound staleness by construction; the authors say their result offers a cheap way to widen that bound.The related-work passage names Fu et al. (2025), Lu et al. (2025), and Li et al. (2027).Source: §5 · p. 9

6 DISCUSSION

Cooling sharpens the sampler's proposal, changing the mismatch the learner must correct; its benefit depends on staleness and has limits across temperature, task, and training stage.

6 DISCUSSION

  • Cooling proposes responses the learner already rates highly, whose learner–sampler ratio drifts more slowly than for low-probability tokens added by hotter sampling.The effect scales with mismatch: it makes no difference at N=96, distinguishes collapse from stability at N=192, and is outgrown at N=288.Source: §6 · p. 9
  • Temperature and refresh interval form one design decision: cooling helps under staleness, but its effect depends on the mismatch and there is no universal temperature.The paper contrasts stale sampling with on-policy results, where Liu et al. (2025) find Tb=0.6 unstable and Tb=1.2 safe.Source: §6 · p. 9
  • The evidence covers one model family, two mathematical tasks, and 300 steps; the 7B comparison has two data seeds per arm.Staleness is emulated with a fixed refresh interval, while a production queue has a lag distribution whose tail matters.Source: §6 · p. 9

7 CONCLUSION

Importance-corrected GRPO reaches a staleness limit on both tested model sizes, while cooling extends usable refresh intervals and preserves progress within a temperature window. The paper also describes its implementation and planned release of logs, curves, and reproduction scripts.

7 CONCLUSION

  • Cooling extends the usable refresh interval while keeping the short interval's learning progress at a fixed update budget, but the benefit depends on temperature, task, and training stage.The conclusion calls temperature and refresh interval one design decision.Source: §7 · p. 9
  • The paper reports a reproducible staleness cliff for importance-corrected GRPO at moderate refresh intervals on 1.5B and 7B models.The within-cycle KL is identified as the cliff's signature.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

  • The authors say they will release all-run training logs, validation curves, and scripts that produce every figure and table.Section 2 gives hyperparameters, staleness emulation, correction, and temperature decoupling; appendices describe implementation patches and run settings.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADAPTIVE TEMPERATURE CONTROL

The adaptive controller tracks how often prompt groups have mixed rewards and uses temperature probes to choose adjustments. Its rules include bounds, a cooldown, refresh resets, and a KL-based gate.

A ADAPTIVE TEMPERATURE CONTROL

  • The controller tracks an exponential moving average of mixed-reward groups across 14 of 16 prompts and probes the remaining two at Tb ± 0.1.When the mixed fraction falls below 0.55, it moves Tb by 0.05 toward the probe with more mixed groups, subject to an eight-step cooldown.Source: A ADAPTIVE TEMPERATURE CONTROL · p. 11
  • Temperature moves are bounded to [0.6, 1.2], reset to 0.8 at each refresh, and blocked away from 1.0 while KL exceeds 0.3.The KL condition uses a moving average.Source: A ADAPTIVE TEMPERATURE CONTROL · p. 11
  • At N=192, the controller heated to 1.05 in the second cycle and the run collapsed at steps 290–300 with maximum KL 0.43.At N=96, it held 0.8 for the first cycle and heated to 1.0–1.05 in later cycles.Source: A ADAPTIVE TEMPERATURE CONTROL · p. 11

B ALL REALIZATIONS OF THE 7B ARMS AT REFRESH INTERVAL 144

At refresh interval 144, the uncooled 7B runs degrade before their first refresh, while cooled runs reach it intact; cycle-aligned readings show that later cycles are not generally harder for cooled runs.

B ALL REALIZATIONS OF THE 7B ARMS AT REFRESH INTERVAL 144

  • In cooled runs, learner–sampler mismatch at cycle ages 55–65 is lower in cycle two than cycle one, except for seed 43, whose mismatch grows fastest.The seed-43 run also starts its second cycle from the lowest entropy; whether the sharper refreshed policy drives its faster mismatch growth remains open.Source: B ALL REALIZATIONS OF THE 7B ARMS AT… · p. 11, Table 3 · p. 12
  • At refresh interval 96, adaptive controllers and a linear temperature schedule finish within 0.01 of fixed cooling and uncooled TIS; at 192, the controller heats to 1.05 and collapses.A doubled learning rate reaches 0.85 earlier but has one transient collapse at step 200.Source: Figure 7 · p. 12
  • Table 3 reports cycle-aligned KL, entropy, and length-limit clipping, with KL and clipping averaged at cycle ages 55–65 and entropy over the first five cycle steps.Its readings compare cycle one with cycle two and include cycle starts.Source: Table 3 · p. 12
  • Both uncooled 7B runs degrade severely before their first refresh, while no cooled realization degrades before it; two cooled realizations show a late-second-cycle drop.The seed-43 drop begins at step 230 and recovers at the refresh at step 288; the first seed-1 attempt drops at step 280.Source: Figure 8 · p. 12

C IMPLEMENTATION

The implementation uses small verl patches for staleness gating, decoupled sampling temperature, and per-prompt temperatures, plus a µ-GRPO loss port; the reported runs also carry stated compute costs.

C IMPLEMENTATION

  • The experiments use verl with the Section 2 GRPO recipe and three patches that leave the loss unchanged, plus a fourth patch adding the Section 4 µ-GRPO loss.Source: C IMPLEMENTATION · p. 13
  • The staleness gate skips learner-to-vLLM weight synchronization except every N-th training call, while validation always synchronizes fresh weights.Training samples can therefore use a sampler up to N −1 steps old; the first N steps sample from the initial policy.Source: C IMPLEMENTATION · p. 13
  • Decoupled temperature keeps learner, old-policy, and reference passes at temperature 1 while sampling uses Tb and processed log-probabilities record the actual tempered behavior distribution.Raw log-probabilities would make the importance weight πθ/q wrong by a factor that grows with sharpening; short runs found only processed mode yielded the Section 3 behavior.Source: C IMPLEMENTATION · p. 13
  • Per-prompt temperatures are implemented by cloning sampling parameters per request and mapping each rank’s contiguous prompt block by rank and position.A plumbing check found near-duplicate responses only in the half sampled at temperature 0.05, not the half at 1.2.Source: C IMPLEMENTATION · p. 13
  • The µ-GRPO port uses the recipe’s token-mean aggregation and reference KL penalty, but differs from the original system’s response normalization and training setup.The comparison tests the update rule under a common recipe, not a reproduction of the original system; its truncated importance multiplier is off.Source: C IMPLEMENTATION · p. 13
  • A 300-step 1.5B run takes about 3.2 hours on four 32 GB GPUs, a 7B run about 9 hours on four 96 GB GPUs, and 55 Appendix D runs total roughly 220 GPU-box-hours.MATH 1.5B runs take about 3.6 hours.Source: C IMPLEMENTATION · p. 13

D FULL RESULTS LEDGER

The full-results ledger covers every 300-step run and records validation accuracy, degradation timing, and learner–sampler KL outcomes.

D FULL RESULTS LEDGER

  • Table 4 lists each 300-step run’s peak and final validation accuracy, degradation onset and severity, first KL-above-1-nat step, and maximum KL.Validation accuracy is on GSM8K test or MATH-500; onset is defined as at most the running maximum minus 0.10, and severe degradation as at most the maximum minus 0.15.Source: Table 4 · p. 13, D FULL RESULTS LEDGER · p. 14
Loading 2609.36953v1…