PaperlayerView exact revision on arXiv

Source-linked AI summary

Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL

Ziyuan Yang, Yike Wang, Shangbin Feng, Yulia Tsvetkov

arXiv:2609.33126v1cs.LGcs.CL

TL;DR

Group-relative RL can lose its learning signal when all sampled responses receive equally high rewards, leaving increasingly capable models with saturated training data. This paper compares interventions at four GRPO pipeline stages to recycle such data. Negative rollouts are the most effective tested intervention, improving standard GRPO by 9.0% on Qwen3-1.7B and 6.4% on Qwen3-4B.

  • Problem

    When all sampled responses receive equally high rewards, group-relative advantages vanish, raising the question of how to reuse saturated data as models improve.

    Source: §1 · pp. 1–2

  • Method

    The paper tests data-, rollout-, reward-, and advantage-level interventions for recovering useful group-relative signals from saturated examples.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    Negative rollouts improve standard GRPO by 9.0% on Qwen3-1.7B and 6.4% on Qwen3-4B, with the strongest overall performance among tested strategies.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Saturated data can remain useful when paired with semantically meaningful negative trajectories that reveal plausible reasoning failures.

    Source: §7 · p. 10

    Experiments mainly cover mathematical and reasoning benchmarks with verifiable rewards, so transfer to open-ended tasks with harder-to-verify correctness remains unclear.

    Source: A LIMITATIONS · p. 15

Abstract

from arXiv · show

Group-relative reinforcement learning (RL) relies on reward variation among sampled responses to estimate informative relative advantages. As language models become increasingly capable, existing training data can become reward-saturated: all sampled responses to the same problem might receive equally high rewards, where the group-relative learning signals vanish and leave previously useful data obsolete. In this work, we investigate whether useful learning signals can be recovered from such saturated data. We study interventions at four levels of group-policy RL pipelines---data, rollout, reward, and advantage---and conduct extensive RL training on saturated reasoning data only. While standard GRPO on saturated data would almost always yield near-0 advantages and near-noise signals, diverse interventions successfully recycle and repurpose such data: among the proposed strategies, interventions at rollout generation are consistently most effective: nudging the policy to generate ``high-quality'', incorrect solutions introduces rollouts with poor rewards into saturated groups as negative samples, which turns out to improve GRPO by 6.4% to 9.0% across Qwen3-1.7B and 4B. Other interventions such as increasing rollout temperature or adding auxiliary rewards can also restore non-zero advantages, but yield less consistent gains. Further analyses show that effective negative rollouts require informative negative trajectories, that the method remains effective alongside unsaturated data, and that it supports iterative recycling of newly saturated examples. While increasingly stronger LLMs would render more data as saturated, our results demonstrate that don't waste your saturated data: with the right strategies they can be recycled into useful RL training signals in an increasingly data-scarce world.

1 INTRODUCTION

Group-relative learning signals vanish when sampled responses all receive uniformly high rewards, a regime the paper calls reward saturation. The paper asks whether saturated data can be recycled and compares interventions across four GRPO stages, finding negative rollouts most effective.

1 INTRODUCTION

  • As problems become easy, uniformly high rewards for correct rollouts make group-relative learning signals vanish, a concern as LLM progress outpaces challenging training data.Creating harder problems requires substantial expert effort, reliable verification, and careful distribution control.Source: §1 · pp. 1–2
  • The study tests interventions at the data, rollout, reward, and advantage stages using saturated data across models and eight datasets spanning math, reasoning, and instruction following.Examples include adding irrelevant context, constructing incorrect rollouts, adding fine-grained rewards, and modifying advantage estimation.Source: §1 · p. 2
  • Negative rollouts improve standard GRPO by 9.0% on Qwen3-1.7B and 6.4% on Qwen3-4B, outperforming other tested interventions.Higher-temperature sampling, reward shaping, and advantage manipulation can also recover signals, but gains are smaller or less consistent.Source: §1 · p. 2
  • Negative rollouts work best when their unsuccessful trajectories are informative; the approach also works with mixed saturated and unsaturated data and supports iterative recycling.Changing the advantage function alone does not improve performance.Source: §1 · p. 2
  • The results suggest saturated examples can retain utility for RL as stronger models cause more data to become reward-saturated.The paper frames recycling as an alternative to discarding such data.Source: §1 · p. 2

2 RELATED WORK

Prior group-based RL work studies how to recover learning signals when rollouts are uniformly poor or when prompts become reward-saturated. This paper positions its contribution as a systematic comparison of interventions across four RL stages.

2 RELATED WORK

  • Group-based methods such as GRPO and DAPO estimate relative advantages from multiple responses to one prompt, relying on within-group reward variation.Identical rewards collapse group-relative advantages.Source: §2 · pp. 2–3
  • Prior work on low-information groups focuses mainly on uniformly incorrect rollouts, using filtering, exploration, advantage changes, curricula, or harder-problem construction.The paper instead studies prompts whose rollouts are successful but whose relative advantages collapse.Source: §2 · p. 3
  • Recent saturation mitigations avoid uninformative groups, replay effective groups, or restore reward variation through virtual samples, trajectory-quality signals, or constrained exploration.The paper extends this line by comparing interventions at the data, rollout, reward, and advantage stages.Source: §2 · p. 3

3 METHOD

The method first characterizes GRPO’s dependence on reward variation within a prompt’s rollout group, then tests interventions at four pipeline stages to recover signals from saturated groups.

3.1 GRPO PRELIMINARY

  • GRPO samples multiple rollouts per prompt, assigns binary verifiable rewards, computes group-relative advantages, and optimizes with a clipped objective and KL regularization.The advantage is normalized using the group reward mean and standard deviation.Source: §3.1 · p. 3

3.2 INTERVENTION AT DIFFERENT LEVELS

  • A group is reward-saturated when all responses receive the maximum reward, causing reward variance and group-relative advantages to vanish.The paper identifies saturated prompts by sampling G responses and retaining groups in which all are correct.Source: §3.2 · p. 3
  • Data-level interventions add irrelevant information or rephrase prompts to induce more varied rollout outcomes and within-group rewards.The study tests whether these perturbations reduce saturation.Source: §3.2 · p. 4
  • Rollout-level interventions increase sampling temperature or construct incorrect negative rollouts for groups whose sampled responses are all correct.Negative rollouts are requested from the current policy by appending an instruction to generate an incorrect solution, then grouped with correct rollouts.Source: §3.2 · p. 4
  • Reward-level interventions add one auxiliary reward at a time—reward-model score, reasoning quality, or response diversity—to binary verification reward.The auxiliary signal is weighted by λ to test whether finer reward granularity restores informative group-relative advantages.Source: §3.2 · p. 4
  • Zero-Padding adds k zero-reward entries when computing group statistics, producing non-zero advantages for successful rollouts without changing their shared positive preference.It increases optimization-signal magnitude without introducing new information or changing data, trajectories, or the reward function.Source: §3.2 · pp. 4–5

4 EXPERIMENT

Experiments train on a filtered MATH subset of reward-saturated problems and evaluate generalization across eight benchmarks. Comparisons include standard GRPO and other pretrained, RL, and supervised-training baselines.

4.1 EXPERIMENT SETTINGS

  • Training uses a 4.3K MATH subset retaining problems for which all eight base-model rollouts are correct, with evaluation on eight math, reasoning, and instruction-following benchmarks.The benchmarks include MATH-500, Minerva, BBH, GPQA-Diamond, AIME24, AIME25, IFBench, and IFEval.Source: §4.1 · p. 5
  • The primary backbone is Qwen3-1.7B and 4B, trained for two epochs with group size G = 8; negative-rollout groups use Gwrong = 2 while keeping total group size fixed.The experiments use the verl framework on NVIDIA H200 GPUs.Source: §4.1 · p. 5

4.2 BASELINES

  • Baselines include the base model, standard GRPO, Mixed CUTS, teacher-solution SFT, and SFT followed by GRPO.Mixed CUTS constructs exploratory rollouts after warmup tokens; the teacher is Qwen3-8B.Source: §4.2 · p. 5

5 RESULTS

On saturated data, negative rollouts yield the strongest reported aggregate performance, improving average scores over standard GRPO at both Qwen3 sizes. Diversity perturbations and reward- or advantage-level interventions can recover signals but are less consistent.

5 RESULTS

  • Standard GRPO improves saturated-data averages modestly, while SFT and SFT→GRPO fall short of GRPO and Mixed-CUTS varies by model size.Mixed-CUTS reaches 30.91 on Qwen3-1.7B but 36.79 on Qwen3-4B, below GRPO on the larger model.Source: §5 · p. 5, Table 1 · p. 6
  • +9.0% on Qwen3-1.7B and +6.4% on Qwen3-4B: negative rollouts improve average scores over standard GRPO at both model sizes.Scores rise from 28.93 to 31.53 on Qwen3-1.7B and from 38.74 to 41.22 on Qwen3-4B.Source: §5 · p. 6, Table 1 · p. 6
  • Temperature and irrelevant-context perturbations vary by scale: temperature raises Qwen3-1.7B from 28.93 to 29.74 but lowers Qwen3-4B from 38.74 to 38.05; irrelevant information helps only Qwen3-4B.Prompt rephrasing underperforms GRPO at both scales; the results point to informative within-group reward differences, rather than diversity alone.Source: §5 · pp. 6–7
  • Reward shaping and advantage manipulation can restore weaker signals, but their gains are less consistent than negative rollouts.Reward-model supervision reaches 40.24 on Qwen3-4B, below negative rollouts at 41.22; zero-reward padding helps Qwen3-1.7B but substantially underperforms GRPO on Qwen3-4B.Source: §5 · p. 7

6 ANALYSIS

Across its analyses, the paper finds that negative rollouts work best when they form coherent unsuccessful trajectories, remain effective with mixed saturated and unsaturated data, and can be reused as saturation grows. It also finds that larger advantages alone do not improve learning, while negative rollouts transfer to another model family.

6.1 EFFECTIVE NEGATIVE ROLLOUTS REQUIRE INFORMATIVE TRAJECTORIES

  • 31.53 on Qwen3-1.7B and 41.22 on Qwen3-4B are the strongest average scores for model-generated negative rollouts among the three tested strategies.Answer replacement scores 27.70 and 33.38; wrong-answer-conditioned continuation scores 30.29 and 37.31, respectively.Source: Table 2 · p. 7, §6.1 · p. 7

6.2 NEGATIVE ROLLOUTS GENERALIZE BEYOND FULLY SATURATED DATA

  • 1.81–3.09 points (4.8–8.3% relative) are the average gains over GRPO across tested saturation ratios.At 40% saturation, average performance rises from 37.97 to 39.78; Negative Rollout is applied only to saturated groups.Source: Figure 2 · p. 8, §6.2 · p. 8

6.3 TOWARDS RSI-STYLE SELF-EVOLVING SATURATION-AWARE TRAINING

  • 443 newly saturated prompts are added by step 80 as the policy improves, while AIME 2025 validation accuracy rises from 0.2208 to 0.3271.The paper calls this iterative procedure an RSI-style self-evolving saturation-aware training loop and describes the experiment as a preliminary realization.Source: Figure 3 · p. 8, §6.3 · p. 8

6.4 A LARGER ADVANTAGE DOESN’T LEAD TO BETTER PERFORMANCE

  • Zero-reward padding increases correct-rollout advantages monotonically with k, but performance does not improve monotonically and all tested variants underperform standard GRPO.The added contrast is numerical and supplies no additional trajectory-level information.Source: Table 3 · p. 9, §6.4 · p. 9

6.5 GENERALIZATION ACROSS MODEL FAMILIES

  • 25.63 average score on LLaMA-3.1-8B-Instruct with negative rollouts exceeds GRPO’s 24.97, while Mixed-CUTS scores 17.35.The cited analysis reports pronounced gains on Minerva (13.97→20.59), GPQA (20.71→24.75), and AIME24 (3.30→10.00).Source: Table 4 · p. 9, §6.5 · p. 9

7 CONCLUSION

In GRPO, reward saturation leaves little relative advantage signal, but constructing wrong-answer rollouts can recover useful learning signals. The conclusion reports consistent gains across Qwen3-1.7B and Qwen3-4B and transfer to LLaMA3.1-8B-Instruct, while identifying limits in task scope, model coverage, negative-trajectory quality, and iterative mining.

7 CONCLUSION

  • Constructing wrong-answer rollouts at the rollout level most effectively recovers useful learning signals from saturated GRPO examples, consistently improving average performance across Qwen3-1.7B and Qwen3-4B.The approach also transfers to LLaMA3.1-8B-Instruct.Source: §7 · pp. 9–10
  • Saturated examples may remain useful when paired with semantically meaningful negative trajectories that reveal plausible reasoning failures.Changing reward statistics, generic perturbations, or teacher imitation is less reliable.Source: §7 · p. 10

AI USE STATEMENT

  • The authors report using generative AI to aid and polish writing, but not to design methodology, conduct experiments, or generate synthetic datasets.They take responsibility for the final content, including AI-aided text, claims, or artifacts.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The experiments use public mathematical and general reasoning benchmarks, involve no human subjects or private data, and use model-generated negative trajectories solely as training signals.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

A LIMITATIONS

  • The experiments mainly cover mathematical and reasoning benchmarks with verifiable rewards; transfer to open-ended tasks with harder-to-verify correctness remains unclear.Systematic experiments use Qwen3 and LLaMA3.1; broader scaling studies across larger model families are needed.Source: A LIMITATIONS · p. 15
  • The method depends on the quality and diversity of generated incorrect trajectories, since poor negatives may add noise or encourage superficial contrast.Iterative saturation mining results are preliminary and cover only a limited number of steps.Source: A LIMITATIONS · p. 15

B EXPERIMENTAL DETAILS

The experimental details specify prompts for standard reasoning and deliberately incorrect solutions, alongside GRPO settings and intervention-specific configurations. The wrong-answer prompt asks the model to follow a plausible mathematical misconception without naming it.

B.1 SYSTEM PROMPT

B.2 WRONG ANSWER PROMPT

  • The wrong-answer prompt asks for a coherent solution that follows one subtle, plausible mathematical misconception and ends with an incorrect boxed answer.The misconception is chosen privately and not explicitly mentioned.Source: §B.2 · p. 15

B.3 HYPERPARAMETER SETTINGS

  • GRPO uses AdamW with learning rate 1 × 10−6, weight decay 1 × 10−2, batch size 128, response length 4096, group size G = 8, and KL coefficient β = 1 × 10−3.Unless specified otherwise, temperature is 1.0 and training lasts 2 epochs.Source: §B.3 · p. 15, Table 5 · p. 16
  • The higher-temperature variant uses t = 1.5, negative rollout uses Gwrong = 2 trajectories per saturated group, and iterative mining resamples every 10 training steps.Reward interventions use Skywork-Reward-V2-Qwen3-8B and Qwen3-8B as reward model and LLM judge, respectively.Source: §B.3 · p. 15, Table 5 · p. 16

C ROBUSTNESS ACROSS RANDOM SEEDS

Across three random seeds on Qwen3-1.7B, Negative Rollout consistently outperforms standard GRPO, with a higher average score and gains across each run.

C ROBUSTNESS ACROSS RANDOM SEEDS

D FURTHER ANALYSIS OF SUPERVISED FINE-TUNING

SFT and SFT→GRPO do not consistently outperform standard GRPO, whereas Negative Rollout achieves stronger aggregate performance. SFT changes predictions substantially, but positive-only teacher trajectories provide limited information about which successful behaviors to strengthen or avoid.

D FURTHER ANALYSIS OF SUPERVISED FINE-TUNING

  • On MATH, SFT fixes 25 previously incorrect examples and breaks 25 correct ones; on GPQA, it fixes and breaks 51 each; on AIME24, it fixes 33 and breaks 67.Source: D FURTHER ANALYSIS OF SUPERVISED… · p. 16
  • SFT and SFT→GRPO do not consistently outperform standard GRPO, while Negative Rollout achieves stronger aggregate performance.Source: D FURTHER ANALYSIS OF SUPERVISED… · p. 16
  • Positive-only teacher imitation offers limited information about behaviors to strengthen or avoid when the model already produces successful trajectories; Negative Rollout contrasts successful and unsuccessful trajectories for the same prompt.Source: D FURTHER ANALYSIS OF SUPERVISED… · p. 16

E.1 FULL DERIVATION

The derivation starts with a group of G correct rollouts, whose rewards have zero variance and yield no useful relative advantage. It appends k zero-reward samples and derives the resulting normalized advantages.

E.1 FULL DERIVATION

  • Appending k artificial zero-reward samples to G correct rollouts gives the group reward variation needed to derive normalized advantages.The derivation uses n = G + k and sample variance with denominator n − 1.Source: §E.1 · p. 17
  • The derivation gives separate normalized advantages for original reward-1 samples and added zero-reward samples.The group mean and sample variance are used before deriving each sample type’s advantage.Source: §E.1 · p. 17
  • The per-correct-sample advantage A1(k) increases monotonically with k, so maximizing it alone would favor arbitrarily many zero-reward samples.The monotonicity holds for G > 1 and k ≥ 0.Source: §E.1 · p. 17
Loading 2609.33126v1…