PaperlayerView exact revision on arXiv

Source-linked AI summary

RLoop: An Self-Improving Framework for Reinforcement Learning with Iterative Policy Initialization

Zeng Zhiyuan, Jiashuo Liu, Zhangyue Yin, Ge Zhang, Wenhao Huang, Xipeng Qiu

arXiv:2511.04285v1cs.AI

TL;DR

RLVR can show rising training reward while generalization stagnates or degrades, and training can forget earlier capabilities. RLoop cycles RL exploration and RFT consolidation; the authors report better performance than vanilla RL, particularly in pass@k.

  • Problem

    RLVR can improve training reward while out-of-distribution accuracy and pass@k stagnate or degrade, raising concern about generalization.

    Source: §1 · p. 1

  • Method

    RLoop alternates RL exploration for diverse solutions with RFT on successful trajectories to refine the policy for the next iteration.

    Source: §1 · pp. 1–2

  • Results

    RLoop significantly outperforms vanilla RL, particularly in pass@k, while mitigating long-term forgetting and improving solution diversity and training stability.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The authors frame RL training instability as a source of exploration for more stable, generalizable reasoning models.

    Source: §6 · p. 9

    The ideal expert distribution is unknown, so the stated objective of matching it cannot be directly specified from that distribution.

    Source: §4.3 · p. 5

Abstract

from arXiv · show

While Reinforcement Learning for Verifiable Rewards (RLVR) is powerful for training large reasoning models, its training dynamics harbor a critical challenge: RL overfitting, where models gain training rewards but lose generalization. Our analysis reveals this is driven by policy over-specialization and catastrophic forgetting of diverse solutions generated during training. Standard optimization discards this valuable inter-step policy diversity. To address this, we introduce RLoop, a self-improving framework built on iterative policy initialization. RLoop transforms the standard training process into a virtuous cycle: it first uses RL to explore the solution space from a given policy, then filters the successful trajectories to create an expert dataset. This dataset is used via Rejection-sampling Fine-Tuning (RFT) to refine the initial policy, creating a superior starting point for the next iteration. This loop of exploration and exploitation via iterative re-initialization effectively converts transient policy variations into robust performance gains. Our experiments show RLoop mitigates forgetting and substantially improves generalization, boosting average accuracy by 9% and pass@32 by over 15% compared to vanilla RL.

1. Introduction

RLVR training can improve reward while generalization stagnates or degrades, alongside forgetting of earlier capabilities. RLoop alternates RL exploration with RFT consolidation and improves on vanilla RL in challenging math reasoning benchmarks.

1. Introduction

  • RL overfitting appears when reward keeps rising while out-of-distribution accuracy and pass@k stagnate or degrade.The reported example shows reward increasing for over 700 steps while generalization performance stagnates or degrades around 140 steps.Source: §1 · p. 1
  • Standard RL can discard about 30% of knowledge acquired early, and policies at different training steps provide underused diversity.The authors link catastrophic forgetting to overfitting and characterize inter-step policy diversity as a potential exploration resource.Source: §1 · pp. 1–2
  • RLoop alternates RL generation of diverse trajectories with filtering successful outcomes and using the resulting expert data for RFT.The refined policy becomes the starting point for the next exploration phase, turning the process into iterative policy initialization.Source: §1 · pp. 1–2
  • RLoop significantly outperforms vanilla RL on challenging math reasoning benchmarks, particularly in pass@k, while mitigating forgetting and improving training stability.The introduction reports these outcomes without specifying quantitative gains.Source: §1 · p. 2

2. Related Works

Prior work addresses RLVR generalization through data, algorithm, or initialization changes, while RL–SFT methods range from pipelines to joint or interleaved training. RLoop differs by iterating between RL and RFT while bootstrapping its own expert data.

2. Related Works

  • Prior RLVR generalization findings conflict: some studies report gains, while others find pass@k degradation or improvements limited to certain tasks.Yue et al. (2025) report greedy-accuracy gains but pass@k degradation, especially for large k; Liu et al. (2025b) report pass@k gains on some tasks.Source: §2 · p. 2
  • Generalization approaches enrich data, modify RL algorithms, or improve initialization through reasoning-focused pretraining or fine-tuning.Examples include question augmentation and curricula, entropy- or variance-based gradient methods, and high-quality reasoning corpora.Source: §2 · pp. 2–3
  • RL–SFT combinations include sequential pipelines, off-policy or joint-loss integration, interleaved SFT, and alternating reward-model and policy updates.The cited examples include Ouyang et al. (2022), Yan et al. (2025), Zhang et al. (2025b), Chen et al. (2025), Ma et al. (2025a), Yuan et al. (2025), Chen et al. (2024), and Zhong et al. (2025).Source: §2 · p. 3
  • RLoop uses cyclical, macrolevel RL and RFT phases and bootstraps its expert data from successful RL explorations rather than external expert data.The authors distinguish this self-improving initialization loop from methods using external data or extensive human curation.Source: §2 · p. 3

3. Preliminary Study: Characterizing Policy Dynamics in RLVR

The preliminary study tracks learning, forgetting, and trajectory similarity across RL checkpoints. It finds continued capability acquisition alongside substantial forgetting, while solution similarity declines as checkpoints grow farther apart.

3. Preliminary Study: Characterizing Policy Dynamics in RLVR

  • Learning rate measures newly solvable validation problems, forgetting rate measures lost solvable problems, and trajectory similarity tracks solution-form shift across steps.The analyses sample 32 solutions per validation question.Source: §3.1 · p. 3, §3 · p. 3, Figure 2 · p. 4

3.1. Learning and Forgetting Dynamics

  • Later checkpoints solve validation problems earlier checkpoints could not, indicating that training continues to add problem-solving capabilities.The Learning Matrix’s non-zero upper-triangle values represent this acquisition.Source: §3.1 · p. 3, Figure 2 · p. 4
  • Forgetting rates frequently exceed 10% and reach 35%, while learning initially exceeds forgetting but later may be matched or surpassed by it.This changing balance is offered as an explanation for validation performance oscillation or decline and why prolonged training need not improve generalization.Source: §3.1 · pp. 3–4, Figure 2 · p. 4

3.2. Similarity Matrix

  • Intra-step trajectory similarity is typically above 0.26, versus below 0.2 across steps, and decreases with checkpoint distance.The study measures bi-gram overlap using pairwise Jaccard similarity aggregated across solution pairs.Source: §3.2 · p. 4

4. Methodology

RLoop alternates RL exploration with RFT consolidation, using each refined policy as the starting point for the next iteration. Its exploitation phase targets hard problems, and its RFT objective is grounded as importance-weighted maximum-likelihood training.

4.1. RLoop Framework

  • RL explores diverse solution modes from intermediate checkpoints; reward-filtered successful trajectories then fine-tune the initial policy, whose refinement starts the next iteration.The RL phase is used as a search process rather than trained to convergence.Source: §4.1 · p. 4

4.2. Active Learning for Focused Exploitation

  • RFT may over-invest in consistently solved problems, so active learning restricts updates to successful trajectories from problems the current policy finds hard.This focuses capacity on expanding capabilities at the knowledge frontier and avoids redundant updates on mastered tasks.Source: §4.2 · p. 5

4.3. Theoretical Grounding: RFT as Importance-Weighted MLE

  • For binary rewards, reward approximates the unknown importance weight, making RFT’s objective equivalent to SFT on the rejection-sampled expert dataset.The paper grounds RFT as importance-weighted maximum-likelihood estimation.Source: §4.3 · pp. 5–6

5. Experiments

RLoop outperforms vanilla RL across the reported benchmarks, while its repeated RL-and-RFT cycle is associated with less long-term forgetting, sustained solution diversity, and greater training stability. The experiments also examine the setup and performance across iterations.

5.1. Experiment Setting

  • Training uses DAPO-17k and Qwen-2.5-7b-Math; each RLoop iteration runs 200 RL steps, then one RFT epoch on successful trajectories from problems with under 10% success.The vanilla RL baseline runs 600 steps, comparable in training budget to three RLoop iterations because RFT overhead is negligible.Source: §5.1 · p. 6, §5.2 · p. 6

5.2. Main Results

  • RLoop outperforms vanilla RL across all evaluation benchmarks on Average@32 accuracy and Pass@k; vanilla RL has lower Pass@k than the base model on three benchmarks, while RLoop surpasses it.The AIME 2024 gain is particularly pronounced, and AIME 2024 was used for checkpoint selection.Source: §5.2 · pp. 6–7

5.3. Scalability Analysis

  • RLoop performance improves across iterations on accuracy and Pass@k, with more pronounced Pass@k gains; unlike vanilla RL, it continues reaching new highs as training proceeds.Vanilla RL validation performance peaks around 300 steps and then declines, while each RFT phase gives RLoop a superior starting point.Source: §5.3 · pp. 7–8, Figure 3 · p. 7, Figure 4 · p. 7

5.4. Why can RLoop Improve Generalization of RL?

  • RLoop generally forgets less than vanilla RL, especially between iterations, although forgetting within each 200-step RL phase remains comparable.The authors attribute the reduced between-iteration forgetting to RFT consolidating knowledge and anchoring the policy.Source: §5.4 · p. 8
  • RLoop maintains lower n-gram similarity than vanilla RL, indicating more diverse solutions, while keeping policy entropy comparable.The authors identify this diversity as a factor contributing to improved generalization and Pass@k.Source: §5.4 · p. 8, Figure 5 · p. 8

5.5. RLoop Improves Training Stability

  • Vanilla RL’s gradient norm grows explosively after about 600 steps and training collapses, whereas RLoop stays below 0.3 through 800 RL steps.The paper attributes RLoop’s stability to shorter RL explorations and periodic re-anchoring through fine-tuning.Source: §5.5 · p. 9, Figure 6 · p. 9

6. Conclusion

The conclusion frames RL overfitting as linked to catastrophic forgetting and underused inter-step policy diversity. It presents RLoop’s alternating RL exploration and RFT consolidation as a framework that outperforms vanilla RL, particularly on Pass@k, while mitigating forgetting and improving stability.

6. Conclusion

  • The authors link RL overfitting to catastrophic forgetting and under-utilization of inter-step policy diversity.They identify these as central challenges in RLVR.Source: §6 · p. 9
  • RLoop alternates RL exploration, which generates diverse solutions, with RFT exploitation, which consolidates knowledge.The authors describe the cycle as turning RL’s instability into a strength.Source: §6 · p. 9
  • The experiments show RLoop outperforms vanilla RL, particularly on Pass@k, while mitigating long-term forgetting and ensuring training stability.The authors conclude that the framework offers a path toward more stable, generalizable reasoning models.Source: §6 · p. 9

A. Detailed Calculation of Trajectory Similarity

Appendix A outlines a method for measuring n-gram-based similarity between solution sets generated at two training steps.

A. Detailed Calculation of Trajectory Similarity

1. Pairwise Solution Similarity via Jaccard Index

The paper defines pairwise solution similarity across training steps using shared n-grams and the Jaccard similarity coefficient.

1. Pairwise Solution Similarity via Jaccard Index

  • The method compares one solution from step i with one from step j using their shared n-grams.The pair is drawn from the two steps’ generated solution sets.Source: §1 · p. 12
  • The implementation uses bigrams, with n = 2, to form each solution’s n-gram set.These sets are the inputs to the pairwise similarity calculation.Source: §1 · p. 12
  • The Jaccard score measures shared n-grams relative to the total unique n-grams across both solutions.It expresses the proportion of overlap between the two solutions’ n-gram sets.Source: §1 · p. 12

2. Overall Step Similarity via Averaging

The method computes similarity between two training steps by averaging pairwise solution Jaccard scores within each prompt, then averaging those prompt-level scores across the dataset.

  • For each prompt, the method computes Jaccard similarity for all N × N pairs of solutions from steps i and j.Source: §2 · p. 12
  • The prompt-level similarity Simprompt(Si, Sj) is the arithmetic mean of those N^2 pairwise scores.Source: §2 · p. 12
  • The final reported similarity between steps i and j is the average of their Simprompt scores across dataset prompts.The aggregation is described as providing a robust measure of overall similarity between the two solution sets.Source: §2 · p. 12
Loading 2511.04285v1…