PaperlayerView exact revision on arXiv

Source-linked AI summary

SeRL: Self-Play Reinforcement Learning for Large Language Models with Limited Data

Wenkai Fang, Shunyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, Kaixuan Chen, Mingli Song, Dacheng Tao

arXiv:2505.20347v2cs.CLcs.AI

TL;DR

Specialized-domain RL for LLMs is constrained by scarce high-quality instructions and verifiable rewards. SeRL generates adaptive instructions, estimates rewards through majority voting, and iteratively trains on the resulting data; across reasoning benchmarks, it outperforms self-play baselines and matches or exceeds RL with extensive verifiable-reward data.

  • Problem

    Effective LLM RL commonly requires high-quality instructions and verifiable rewards, but both are difficult and costly to obtain in specialized domains.

  • Method

    SeRL combines adaptive self-instruction with online filtering and majority-voting self-rewarding to perform iterative unsupervised RL from limited initial data.

  • Results

    Across reasoning benchmarks, SeRL outperforms other multi-round self-play methods and reaches or surpasses RL-GT performance across reported model and iteration settings.

  • Takeaways & Limitations

    Limited initial data can support effective iterative RL when instruction generation, difficulty filtering, and reward estimation are performed within the self-play framework.

  • Takeaways & Limitations

    Majority-voting rewards may not apply to tasks without deterministic final answers or where process correctness matters, and reward reliability can weaken on challenging problems.

Abstract

from arXiv · show

Recent advances have demonstrated the effectiveness of Reinforcement Learning (RL) in improving the reasoning capabilities of Large Language Models (LLMs). However, existing works inevitably rely on high-quality instructions and verifiable rewards for effective training, both of which are often difficult to obtain in specialized domains. In this paper, we propose Self-play Reinforcement Learning (SeRL) to bootstrap LLM training with limited initial data. Specifically, SeRL comprises two complementary modules: self-instruction and self-rewarding. The former module generates additional instructions based on the available data at each training step, employing robust online filtering strategies to ensure instruction quality, diversity, and difficulty. The latter module introduces a simple yet effective majority-voting mechanism to estimate response rewards for additional instructions, eliminating the need for external annotations. Finally, SeRL performs conventional RL based on the generated data, facilitating iterative self-play learning. Extensive experiments on various reasoning benchmarks and across different LLM backbones demonstrate that the proposed SeRL yields results superior to its counterparts and achieves performance on par with those obtained by high-quality data with verifiable rewards. Our code is available at https://github.com/wantbook-book/SeRL.

1 Introduction

RL has improved LLM reasoning, but effective training typically requires large, high-quality instructions and verifiable rewards that are difficult to obtain in specialized domains. SeRL addresses this data-scarcity problem through online self-instruction, self-rewarding, and iterative RL.

  • Motivation: High-quality instructions and verifiable rewards support effective RL training for LLM reasoning, but are difficult to collect in specialized domains.Human annotation is labor-intensive, requires substantial expertise, and is difficult to scale; generated data may also require expert-level LLMs.
  • Motivation: Existing self-instruction methods generate static datasets and treat model-generated responses as labels, making them unsuitable for evolving online RL settings.Online training requires data that adapts to changing model capabilities and reliable reward estimation.
  • SeRL: SeRL bootstraps LLM training from limited initial data through autonomous instruction generation and reward estimation.The framework performs unsupervised RL on the generated data.
  • SeRL: Self-instruction generates filtered instructions matched to current model capability, while self-rewarding estimates rewards by majority voting without verifiable labels.The modules target instruction quality, diversity, and difficulty while removing dependence on external supervision.
  • Contributions: Extensive experiments show SeRL outperforms advanced self-play counterparts and approaches the performance of training with full high-quality data and verifiable rewards.The reported comparison covers diverse reasoning benchmarks and limited initial data.

2 Background

The background formalizes LLM reinforcement learning as optimizing rewards for generated responses and introduces the notation used for instruction datasets, states, advantages, and clipped policy updates. SeRL applies this RL setup while generating training instructions from a small seed dataset.

  • Reinforcement Learning for LLMs: LLM RL generates a response sequence for each instruction and optimizes a reward R(x, y) associated with the input-output pair.The paper uses Reinforce++ for robustness and stable performance across diverse tasks.
  • Reinforcement Learning for LLMs: The RL objective uses token-level advantages with a clipped policy ratio, where D is the instruction dataset and πθold denotes the pre-update policy.The clipping hyperparameter ϵ is introduced to stabilize training, and the advantage computation depends on the RL algorithm.
  • Self-Instruction: SeRL starts from a small seed dataset and generates a new batch of instructions at every training step.The generated data are organized across training steps within each iteration.
  • Framework overview: Figure 1 organizes SeRL around self-instruction for generating filtered instructions and self-rewarding for unsupervised RL using majority-voting rewards.These components provide the framework overview used by the later method description.

3 Self-play Reinforcement Learning

SeRL combines adaptive instruction generation, online filtering, majority-voting reward estimation, and iterative RL to train from limited data without verifiable labels. The method filters unsuitable tasks before using sampled responses and consensus rewards for repeated policy updates.

  • 3 Self-play Reinforcement Learning: SeRL expands limited initial data with few-shot instructions and enables unsupervised RL without verifiable labels.Its two modules are self-instruction and self-rewarding.
  • 3.1 Self-Instruction: Self-instruction generates new instructions from initial and recently generated examples so the data distribution and difficulty adapt to evolving model capabilities.Recently generated instructions are selectively reused as few-shot examples for subsequent generation.
  • 3.1 Self-Instruction: The online filter removes redundant, visual, excessively short or long, and improperly difficult instructions using ROUGE-L, keyword, length, and majority-answer-ratio criteria.Retained instructions satisfy γdiff ≤ rmean ≤ γeasy, where rmean is computed from majority-voting responses.
  • 3.1 Self-Instruction: Difficulty filtering avoids all-correct or all-incorrect tasks that provide no useful advantage signal and can otherwise cause reward hacking or model degradation.The paper links extreme cases to zero advantage in Reinforce++ and GRPO.
  • 3.2 Self-Rewarding: Self-rewarding samples multiple responses, identifies the majority answer, and assigns rewards according to consensus, replacing reward models or verifiable labels.If majority answers tie, the shortest response is selected; the online filter ensures a majority can be identified.
  • 3.3 Overall Framework: At each training step, SeRL generates instructions, samples nvote responses, computes rewards, and updates the model before repeating the cycle in later steps and iterations.The resulting instruction-response-reward triplets drive iterative self-improvement.
  • 3.3 Overall Framework: Table 1 compares Pass@1 across benchmarks for the proposed method and baselines under greedy decoding, with bold entries marking the best performance.The table distinguishes the original model as Initial and covers both LLaMA-3.2-3B-Instruct and Qwen-2.5-7B-Instruct settings.

4 Experiments

Across math and general reasoning benchmarks, SeRL uses limited seed data with online filtering and majority-voting rewards to achieve strong iterative RL performance, often matching or exceeding RL-GT and other iterative baselines.

  • Overall Results: After three rounds, SeRL outperforms RL-GT on several benchmarks for both backbones, while LLaMA-3.2-3B-Instruct surpasses RL-GT across nearly all benchmarks after round two.The reported gains include MATH-500, ASDiv, and TabMWP for Qwen-2.5-7B-Instruct after three rounds.
  • Overall Results: Across iterations, SeRL consistently outperforms SR-DPO and I-RPO, whereas both baselines show degradation over repeated training.The paper associates SR-DPO’s weaker results with model-based reward accuracy and hypothesizes that I-RPO’s NLL objective overfits chosen responses, reducing generalization.
  • Generalization: On MMLU-Pro, LLaMA-3.2-3B-Instruct improves across selected categories, eventually surpassing RL-GT in STEM and Humanities, while Social and Other remain largely unchanged.Qwen-2.5-7B-Instruct shows minimal improvement, which the authors attribute to its extensive prior fine-tuning on MATH data.
  • Ablation Study: Without dual-end difficulty filtering, average reward keeps rising while MATH-500 accuracy drops sharply after about 100 steps.The authors identify this divergence as reward hacking caused by unsuitable instruction difficulty.
  • Ablation Study: Removing any online filtering strategy lowers overall performance, and omitting difficulty filtering can trigger reward hacking during training.The ablation compares Pass@1 under matched training steps; the filtering strategies target quality, diversity, and appropriate difficulty.
  • Ablation Study: Majority-voting rewards align more closely with rule-based rewards than model-based rewards, and RL-MV performs comparably to RL-GT across test datasets.LLaMA-3.2-3B-Instruct exceeds RL-GT on several benchmarks, while Qwen-2.5-7B-Instruct surpasses RL-GT on nearly all benchmarks in the reported ablation.

5 Related Work

Prior work studies synthetic instruction generation and self-rewarding to reduce reliance on scarce, expensive supervision, but SeRL combines these ideas for iterative online RL.

  • RL for LLM reasoning depends on high-quality instructions and verifiable labels, which are scarce in specialized domains.
  • Self-instruction methods generate instructions and responses autonomously, but commonly produce fixed datasets that become less suitable as online learners evolve.
  • Self-rewarding methods estimate rewards without verifiable labels, while majority voting uses multiple responses for greater stability and consistency.
  • Unlike TTRL [92], SeRL generates diverse synthetic instructions from limited data and supports multiple iterative policy updates rather than one-round adaptation.

6 Discussion

The discussion identifies ceilings on iterative self-training and limitations of majority-voting rewards, especially when answers or reasoning cannot be reliably aggregated.

  • Ceiling of Self-Iteration: Maj@N establishes a performance ceiling for self-training, and further iterations raise that ceiling only if majority-vote accuracy keeps improving.
  • Ceiling of Self-Iteration: On MATH-500, Maj@16 does not keep improving for LLaMA-3.2-3B-Instruct or Qwen-2.5-7B-Instruct, limiting continual gains.
  • Limitation of Majority-voting Reward: Majority voting may not apply to nondeterministic writing tasks or settings where process correctness matters, such as mathematical proofs.
  • Limitation of Majority-voting Reward: Difficulty filtering removes uncertain instructions as curriculum learning, reducing but not completely resolving bias from unreliable self-rewards on challenging problems.

7 Conclusion

SeRL bootstraps RL from limited initial data through self-instruction, online filtering, and majority-vote self-rewarding, then iteratively trains the model on generated data.

  • SeRL combines self-instruction and self-rewarding to expand limited data, estimate rewards without external labels, and perform iterative standard RL.
  • Its online filtering enforces quality, diversity, and suitable difficulty for generated instructions.
  • SeRL consistently outperforms strong baselines and matches methods trained with large-scale, high-quality labeled data across reasoning benchmarks and LLM architectures.
  • The implementation discussion situates SeRL within RL methods that use KL estimators, including k3 for GRPO and k1 for Reinforce++ and RLOO.

C Implementation Details

The implementation uses Reinforce++ with batch-level return normalization, response-level rewards, online instruction filtering, and specified training configurations for two model backbones.

  • Reinforce++ Algorithm: Reinforce++ computes token-level returns using rewards and a KL penalty against the initial model, with the k1 estimator.
  • Reinforce++ Algorithm: Response-level rewards are assigned only to the final token, simplifying the token-level return computation.
  • Online Instruction Filter: The online filter removes similar, visually dependent, malformed, excessively long or short, and unsuitable-difficulty instructions.
  • Instruction Generation: Few-shot instruction generation samples one-fourth of examples from seed data and three-fourths from previously generated instructions.
  • Training Configuration: The reported configuration uses 16 sampled responses, difficulty thresholds γdiff = 0.2 and γeasy = 0.8, temperature 1.0, and three iterations over 7,500 instructions.
  • SFT Comparison: Self-sampled correct responses produced a consistent SFT performance drop, whereas the experiment also tested higher-quality distilled data for comparison.

D.2 Results of More Iterations

Additional iterations show that SeRL’s performance gradually converges rather than improving indefinitely, while medical experiments test transfer to another data-scarce domain.

  • D.2 Results of More Iterations: Pass@1 results over additional iterations show gradual performance convergence, consistent with the underlying foundation model’s limited capacity.The authors relate this behavior to latent reasoning capabilities already present in pretrained models.
  • D.2 Results of More Iterations: The authors frame SeRL as targeting performance comparable to high-quality data with verifiable rewards, not unlimited improvement through self-play.Unlimited iterative improvement is described as an unrealistic and unsolved problem.
  • D.2 Results of More Iterations: Using 500 MedQA training instructions as seed data, SeRL is compared with reinforcement learning trained on the full 10.2k-example dataset across three medical benchmarks.The evaluation covers MedQA, PubMedQA, and NephSAP.
  • D.2 Results of More Iterations: Self-reward estimates are evaluated against ground-truth rewards using cosine similarity, mean squared error, and mean absolute error.Higher cosine similarity and lower errors indicate closer reward alignment.

E.2 Reward Hacking

SeRL’s dual-end clipped difficulty filtering removes problematic instruction extremes while preserving diversity, helping prevent reward hacking during iterative training.

  • E.2 Reward Hacking: Without dual-end clipped difficulty filtering, training becomes prone to reward hacking, including outputs that derive answers but end with an incorrect final answer.The case study shows a response ending with “The final answer is: 0” despite correct reasoning.
  • E.2 Reward Hacking: The first-round generated dataset D1_gen is analyzed before and after applying the difficulty-filtering strategy.D1_gen denotes the dataset generated during the first iteration.
  • E.2 Reward Hacking: Difficulty filtering reduces both extremely short and excessively long responses, targeting overly easy and overly difficult generated instructions.Response length is used as a proxy for instruction difficulty, with Qwen-3-32B answering generated instructions.
  • E.2 Reward Hacking: A case study examines LLaMA-3.2-3B-Instruct outputs after reward hacking using a rectangular-to-polar-coordinate problem.The example includes the model’s reasoning and final answer behavior.
  • E.2 Reward Hacking: The analysis compares generated-instruction counts before and after filtering, with the y-axis representing the number of instructions.The generated data is organized across generation steps in each iteration.
  • E.2 Reward Hacking: ROUGE-L distributions before and after filtering indicate that generated-instruction diversity is largely preserved while difficulty is controlled.The comparison concerns D1_gen and Dseed distributions.

F Additional Generated Instructions Analysis

Additional analyses find that self-generated instructions resemble the seed and full training data in quality and complexity while remaining diverse in difficulty, syntax, and feature space.

  • F Additional Generated Instructions Analysis: The analysis defines Dseed as 500 examples sampled across MATH difficulty levels and tracks datasets cumulatively generated across the first three training iterations.Dseed supplies the few-shot context for subsequent self-instruction generation.
  • F.1 Length and Quality: Generated instructions have lengths and quality scores comparable to Dseed and the MATH training set.Instruction length is measured with the Qwen-3-32B tokenizer, while quality is rated by Qwen-3-32B after filtering repetitive long instructions.
  • F Additional Generated Instructions Analysis: Figures compare generated-instruction difficulty and diversity, including response-length distributions and feature-space projections.The displayed analyses cover generated-instruction difficulty and diversity across the appendix experiments.
  • F.2 Difficulty: Generated-dataset response lengths are close to those of the MATH training set, suggesting comparable difficulty under the longer-reasoning proxy.Qwen-3-32B generates responses for all instructions, and average response length is compared across datasets.
  • F.3 Diversity: Verb–noun analysis finds diverse and syntactically rich constructions among D1_gen instructions.The Berkeley Neural Parser extracts root-nearest main verbs and their first direct noun objects, summarized through the top 20 verbs and top 4 associated nouns.
  • F.3 Diversity: Most generated instructions have low ROUGE-L similarity with Dseed, indicating novelty rather than simple imitation.The comparison uses ROUGE-L scores between each D1_gen instruction and Dseed.
  • F.3 Diversity: UMAP projections show generated instruction features widely distributed rather than tightly clustered, similarly to the MATH training set.The visualization uses BERT features of D1_gen instructions.

G Additional Analysis of the Self-Iteration Upper Bound

Across multiple iterations, SeRL does not consistently raise Maj@16, suggesting an unchanged performance ceiling, while still improving Pass@1 in data-scarce settings.

  • G Additional Analysis of the Self-Iteration Upper Bound: Maj@16 performance does not exhibit a consistent upward trend after multiple iterations, indicating that the model’s performance ceiling remains unchanged.Maj@16 is treated as an empirical upper bound for Pass@1 under this framework.
  • G Additional Analysis of the Self-Iteration Upper Bound: Despite no incremental improvement in Maj@K, the method achieves substantial Pass@1 gains in data-scarce scenarios through unsupervised reinforcement learning.The authors connect this result to the practical utility of SeRL in specialized domains with limited labeled data.
  • G Additional Analysis of the Self-Iteration Upper Bound: The broader impact discussion identifies potential bias amplification and spurious generalization risks when synthetic data are generated and reinforced without sufficient oversight.These risks are emphasized for high-stakes domains such as clinical decision-making and aerospace control.
Loading 2505.20347v2…