Language Self-Play For Data-Free Training

1 Introduction

• RL still depends on task examples in the form of prompts, leaving LLM training constrained by data availability.Synthetic-data training and more efficient data use are described as responses to this bottleneck.Source: §1 · p. 1
• LSP treats increasingly challenging instruction generation as an agent's actions, dispensing with training from data entirely.The method is framed as a different approach from synthetic data and meta-learning.Source: §1 · p. 1
• One LLM plays Challenger and Solver, improving responses while generating harder instructions and improving the distribution of examples.The two roles are instantiated by one model; the algorithm is called Language Self-Play (LSP).Source: §1 · p. 1, Figure 1 · p. 2
• Experiments report LSP-trained models with just-as-strong or stronger performance than LLMs trained with abundant data.The experiments apply LSP to Llama-3.2-3B-Instruct across benchmark tasks.Source: §1 · pp. 1–2

2.1 Supervised Fine-Tuning

• Supervised fine-tuning maximizes the model's likelihood of available answers, but does not represent answer quality.This limitation motivates the growing use of reinforcement learning during fine-tuning.Source: §2.1 · p. 2

2.2 Reinforcement Learning

• The paper focuses on reinforcement learning methods that need no provided answer examples, with rewards supplied by a verifier or trained reward model.The reward is defined for a query-answer pair.Source: §2.2 · pp. 2–3
• Group-Relative Policy Optimization samples multiple answers to a query and compares them, a technique the paper uses in its algorithm.The group-relative comparison is between answers to the same query.Source: §2.2 · p. 3

3 Language Self-Play

• The Challenger generates queries to minimize task reward while the Solver answers to maximize it, making query generation part of the learning game.The reward may be verification-based or preference-based.Source: §3 · p. 3
• A single language model plays both roles through prompting, with the Challenger generating N queries and the Solver producing G answers for each.The method uses a group-relative trick from GRPO to make the game an efficient RL process.Source: §3 · p. 3
• Group values provide a baseline for Solver response advantages and a measure of query difficulty that Challenger is encouraged to maximize.Challenger is rewarded for queries that probe areas where Solver performance is lacking.Source: §3 · p. 3
• KL regularization constrains deviation from the reference model and prevents Challenger from generating semantically meaningless adversarial sequences; this zero-sum variant is LSP-Zero.The Solver and Challenger losses are added and differentiated to take a gradient step.Source: §3 · p. 4
• Adding reference-model-generated quality self-reward to both players’ rewards makes the game non-zero-sum and supports effectively indefinite self-play training.The self-reward supplements Solver reward and Challenger reward before advantages and losses are computed.Source: §3 · p. 5

4 Related Work

• LLM reasoning is bounded by the intellectual complexity and limited quantity of human-provided prompts, motivating work on alternative training data and methods.Prior work includes synthetic-data bootstrapping and meta-learned data augmentation.Source: §4 · p. 6
• Wu et al. (2024) use self-play for preference maximization on provided prompts, whereas this work uses self-play to stream the learning process.Cheng et al. (2024) study an Adversarial Taboo game, but their approach uses prior playouts from upper-shelf models and game-specific supervised fine-tuning.Source: §4 · pp. 6–7
• Zweiger et al. (2025) autonomously edit data fed to an LLM, but their method still assumes access to training data.The paper states its model is exposed to data only during evaluation.Source: §4 · p. 7
• Prior self-referential algorithms adjust weights or system prompts, while Yuan et al. (2024) introduce self-rewards; LSP combines self-generated data with improved data generation in a competitive game.In LSP, self-rewards act as a regularizer within self-play rather than defining the game alone.Source: §4 · p. 7
• Concurrent autonomous-training methods curate query generation and use majority-voting rewards, specializing in tasks with deterministic, verifiable answers; LSP uses unconstrained queries and neural plus self-rewards for open-ended problems.The paper identifies synthesis of these paradigms as future work.Source: §4 · p. 7

5 Experiments

• The study uses Llama-3.2-3B-Instruct and compares self-play with GRPO trained on Alpaca data, including a self-rewarding ablation.Models are evaluated with sampling temperature τ = 0.01; the self-play comparison also examines LSP as an intermediate stage before RL.Source: §5 · p. 8, §5.1 · p. 8

5.1 The Importance of Self-Rewarding

• On AlpacaEval, data-free LSP-Zero and LSP improve overall on the base model, and LSP achieves a similar overall result to GRPO.LSP performs significantly better than both the base model and GRPO on Koala, a conversational, open-ended instruction dataset.Source: §5.1 · pp. 8–9
• Further RL training raises LSP’s overall win-rate from 36.4% to 39.5%, while LSP-Zero with RL scores 36.3% versus LSP’s 36.4%.Source: §5.1 · p. 9

5.2 Benchmark Evaluation of LSP

• LSP recovers most of GRPO’s performance gains across MATH, GSM8K, HumanEval, and Alpaca without training data; LSP+RL gains are limited on some tasks.MATH and GSM8K use the same standalone LSP model; HumanEval uses MBPP for GRPO training because it has no training set.Source: §5.2 · p. 9, Figure 4 · p. 9

• Language Self-Play combines a framework for perpetual improvement of an LLM and its self-generated data with a practical algorithm for pretrained models.Source: §6 · p. 10
• LSP-Zero and LSP improve pretrained models without training data, especially on conversational tasks.Source: §6 · p. 10
• With preferential rewards, performance is bounded by reward-model judgment and human knowledge of the physical world when ground-truth rewards are unavailable.The authors suggest embodied AI collecting empirical data could expand that knowledge.Source: §6 · p. 10

• The treasure-map example asks for a paper-and-pen map using two rocks, with a grid, marked treasure, landmarks, and symbols.Source: A Example Plays · p. 14
• The bridge example specifies 10 wooden blocks, string, and scissors, then gives construction, testing, and safety instructions.Source: A Example Plays · p. 15
• The assembly example requests subtraction of two signed 16-bit values in 32-bit registers and presents code that places the result in DX.It notes that sign handling must be explicit for smaller word sizes.Source: A Example Plays · p. 16

• The prompt asks for physical properties and possible uses of a large unidentified object without naming it.Source: B An Example Of Unregularized Play · p. 17
• The code sketch defines possible size, shape, material, and color attributes for an unidentified object.Source: B An Example Of Unregularized Play · p. 17
• The example invokes describe, analyze, suggest, and draw operations on an UnknownObject instance.Source: B An Example Of Unregularized Play · p. 17

C Hyper-parameters

• Table 1 lists training hyperparameters for GRPO, LSP-Zero, and LSP.The passage does not provide the hyperparameter values.Source: Table 1 · p. 18