Source-linked AI summary
Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain
Wei Liu, Siya Qi, Yali Du, Yulan He
TL;DR
Existing self-play loops can plateau or collapse because they generate more data without increasing learnable information for bounded observers. The paper reframes self-evolution as a triadic self-synthetic pipeline and proposes asymmetric co-evolution, capacity growth, and proactive information seeking; it concludes that these principles support sustained growth of learnable information, while remaining preliminary and scope-limited.
Problem
Existing self-play systems often plateau, collapse, or produce trivial data because they do not sustain a monotonic increase in learnable information for bounded observers.
Method
The paper models self-evolution as a triadic self-synthetic pipeline and proposes asymmetric co-evolution, growing capacity budgets, and proactive external information seeking.
Results
The paper’s analysis and experiments support framing sustainable self-evolution as increasing learnable information through asymmetric co-evolution, dynamic capacity expansion, and proactive information seeking.
Takeaways & Limitations
Self-evolution should be designed as a dynamic information pipeline rather than a static reinforcement-learning game, with system-level mechanisms that sustain learnable information growth.
Takeaways & Limitations
The approach is preliminary, currently applies asymmetry closing mainly to easy-to-verify domains, and requires final task accuracy alongside learnable-information metrics.
Abstract
from arXiv · showhide
Large language models (LLMs) make it plausible to build systems that improve through self-evolving loops, but many existing proposals are better understood as self-play and often plateau quickly. A central failure mode is that the loop synthesises more data without increasing learnable information for the next iteration. Through experiments on a self-play coding task, we reveal that sustainable self-evolution requires a self-synthesised data pipeline with learnable information that increases across iterations. We identify triadic roles that self-evolving LLMs play: the Proposer, which generates tasks; the Solver, which attempts solutions; and the Verifier, which provides training signals, and we identify three system designs that jointly target learnable information gain from this triadic roles perspective. Asymmetric co-evolution closes a weak-to-strong-to-weak loop across roles. Capacity growth expands parameter and inference-time budgets to match rising learnable information. Proactive information seeking introduces external context and new task sources that prevent saturation. Together, these modules provide a measurable, system-level path from brittle self-play dynamics to sustained self-evolution.
1. Introduction
The paper argues that fragile self-play plateaus because synthetic data does not reliably increase learnable information, and proposes a triadic self-synthetic pipeline for sustained self-evolution.
- Motivation: Self-evolving LLMs act as PROPOSER, SOLVER, and VERIFIER, generating tasks, solutions, and feedback through transformations of the model’s own information.These roles form autonomous self-training loops, with the model serving as the shared information source.
- Motivation: Existing self-play systems often plateau, collapse, or generate trivial tasks without intervention or carefully tuned prompts.Reported failures include identity-like problems, early performance peaks followed by decline, and self-delusion after ground-truth recalibration is removed.
- Central thesis: Sustainable self-evolution requires a self-synthesised data pipeline in which the three roles cooperate to increase learnable information monotonically across iterations.The paper frames this requirement as distinct from fragile self-play dynamics and formalises learnable information under bounded observers.
- Design principles: Asymmetric co-evolution uses divergent synthesis directions to create weak-to-strong supervision, then synchronises solver improvements back into the internal environment as strong-to-weak supervision.The internal environment comprises the PROPOSER and VERIFIER, while the SOLVER is trained through their supervision.
- Design principles: Capacity growth expands parameter and inference-time budgets so the observer can keep pace with increasing learnable information produced by the loop.The proposed capacity budgets cover both model parameters and inference-time computation.
- Design principles: Proactive information seeking acquires external sources aligned with current capabilities, supplying new contexts and synthetic directions that create fresh asymmetries.This design addresses the finite information available to zero-data or fixed-dataset self-evolution.
2. Background
The paper models self-evolution as iterative generation and training on self-synthetic data, then interprets progress through information flow and learnable information relative to bounded observers.
- Triadic self-evolution: At iteration t, one LLM successively proposes tasks, solves them, verifies solutions, and trains on the resulting tasks, solutions, and feedback.The loop has no external labelled dataset, teacher model, or reward model; external context may condition task proposing.
- Information flow: Proposing, solving, and verifying transform information from the pretrained weights into a self-synthesised data stream for the next update.External conditioning can add information absent from the pretrained source, while the loop transforms unlearnable noise into learnable information.
- Learnable information: Learnable information is reusable structure that a learner can capture for improved compression or prediction, whereas unlearnable information remains unpredictable or incompressible under the learner’s assumptions.The distinction is motivated by MDL’s decomposition of description length into model description and prediction loss.
- Bounded observers: Learnable information depends on the observer’s computational limits, so the paper adopts epiplexity to model structure captured under bounded parameter and inference-time budgets.The same data can appear structured to a stronger observer and random to a weaker one.
- Bounded observers: The framework assigns each role iteration-dependent budgets, with C denoting parameter capacity and T denoting inference-time computation.When one budget pair suffices, the notation is written as (C, T).
- Measurement: Epiplexity S_C,T(X) measures reusable structure, while bounded entropy H_C,T(X) represents what remains random to an observer constrained by (C, T).The paper treats epiplexity as a proxy for learnable information and bounded entropy as unlearnable information.
- Measurement: Sustainable progress requires generating data in a Goldilocks zone: complex enough to be non-trivial but structured enough to remain learnable for the current observer.Data that is too simple or too hard provides insufficient useful structure under the observer’s budget.
3. Towards Genuine Self-Evolution
Genuine self-evolution requires a self-synthetic pipeline whose learnable information increases across iterations. Three linked designs—co-evolutionary asymmetry, growing capacity, and proactive information seeking—address the generation, absorption, and renewal of that information.
- Asymmetric Co-evolution: Asymmetric co-evolution uses weak-to-strong supervision from the PROPOSER/VERIFIER to the SOLVER and strong-to-weak synchronisation back to the internal environment.The loop preserves an intelligence gap while transferring improvements across roles.
- Asymmetric Co-evolution: If the internal environment does not co-evolve, an improving SOLVER can face low-structure task and feedback streams that encourage trivial data.Standard reinforcement learning supports the weak-to-strong direction, but reliable gains for the PROPOSER and VERIFIER remain unclear.
- Capacity Growth: Capacity growth expands parameter, inference-time, and training budgets so the observer can absorb increasingly learnable structure.Fixed capacity can saturate training and drive the loop toward tasks that remain easy for the current model class; fixed inference budgets can instead cause truncated reasoning.
- Proactive Information Seeking: Proactive information seeking selects external contexts that generate new synthetic directions, because closed loops and fixed corpora are bounded by existing information.Conditioning on external contexts preserves self-synthetic targets while introducing fresh asymmetries.
- Synergy: Together, the three modules form a single information-production pipeline: asymmetry generates structure, capacity internalises it, and information seeking supplies fresh contexts.Their intended synergy is a monotonic increase in learnable information rather than repeated patterns with limited gains.
4. Experiments
The experiments estimate learnable information with epiplexity and examine how proposer and solver capacities, synthetic directions, and self-play iterations affect it. Stronger proposers can generate more learnable information, but solver capacity has an intermediate optimum and ordinary self-play does not sustain information growth.
- Experimental setup: Epiplexity is estimated with prequential coding by selecting the value at the epoch that minimises the combined MDL score.The procedure measures cumulative online regret relative to final training loss and combines normalised epiplexity with validation loss per token.
- Experiment 1: Experiment 1 compares solver capacities, proposer capacities, and induction, abduction, and deduction synthesis directions.The study uses code-based tasks and synthetic data generated by proposers of varying capacities.
- Experiment 1: Stronger proposers generate synthetic data containing more learnable information, while solver-observed information first increases and then decreases as solver size grows.The solver pattern is attributed to a transition from learning compressible structure to direct memorisation under a fixed computation budget.
- Experiment 1: Induction produces substantially more learnable information than abduction and deduction across the tested synthesis directions.The paper presents this directional difference as evidence that synthetic directions vary in information content.
- Experiment 2: Across repeated self-play iterations, learnable information fluctuates dramatically rather than increasing steadily.The observed dynamics coincide with declining solver capability and collapse of proposer-generated problem patterns.
- Experiment 2: Relying solely on multi-reward reinforcement learning does not achieve sustained evolution without an explicit mechanism that closes the self-training loop.The result supports evaluating self-play by whether its data stream increases learnable information over iterations.
5. Alternative Views
The paper contrasts learnable-information growth with several common explanations of self-evolution, arguing that reward improvement, difficulty, co-evolution, and scaling alone are insufficient criteria. Each perspective captures a useful ingredient but does not by itself ensure expanding learnable structure.
- Self-Evolving via Self-Play RL: Reward optimisation can stabilise training without ensuring that self-synthesised data exposes increasing learnable structure under bounded observers.Reward hacking, memorisation, and multi-reward instability can produce progress signals without the targeted information growth.
- Curriculum Learning for Evolution: Curriculum difficulty does not directly measure whether a new structure becomes learnable, and repeated directions can raise difficulty without information gain.Difficulty may conflate search depth and verification cost.
- Co-Evolution between Agent and Environment: Co-evolution models mutual adaptation but does not distinguish productive adaptation from complexity reshuffling without expanding internal structure.An increasingly challenging environment may therefore fail to expand what the solver can learn.
- Scaling is All you Need: Capacity scaling expands what can be learned but may amplify memorisation unless coordinated with new synthetic directions and preserved asymmetry.The paper treats scaling as necessary alongside information growth rather than sufficient on its own.
6. Related Works
Related work spans self-training, solver–verifier co-evolution, proposer–solver self-play, and triadic loops. These approaches introduce filtering, iterative optimisation, verification, curricula, external corpora, or evolving environments, while the paper focuses on sustained learnable-information growth.
- Self-Training: Early self-training methods recycle filtered or rationalised self-generated traces, but they saturate once the initial data distribution is exhausted.Examples include STaR, ReST, and rejection-sampling scaling studies.
- Solver–Verifier Co-Evolution: Solver–verifier methods iteratively improve responses, generators, policies, or reward models using self-judging, preference optimisation, or verifiable rewards.The cited methods include Self-rewarding, SPIN, Self-boosting, iterative DPO, and URPO.
- Proposer–Solver Self-Play: Proposer–solver self-play systems generate and verify code-based or reasoning tasks using execution, voting, policy optimisation, or unit tests.The cited examples include Absolute Zero, R-Zero, Dr. Zero, and Self-Questioning.
- Triadic Loops: Triadic-loop approaches add curricula, external corpora, frozen teachers, or evolving internal environments to shape task generation and verification.SPELL, SPICE, Socratic-Zero, and GenEnv represent these design choices.
7. Limitations
The framework is preliminary rather than an off-the-shelf solution. Its limitations concern verification scope, the relationship between learnable information and task accuracy, metric validation, and the difficulty of proactive information seeking.
- Scope: Closing the asymmetry gap currently applies only to domains that are easy to verify.The paper calls for more cross-domain and generalisable methods for hard-to-verify settings.
- Evaluation: Learnable information cannot replace final task-accuracy metrics because some learnable information may reflect data structure without helping task completion.The paper recommends considering both metric types together.
- Evaluation: Epiplexity is not yet widely validated, limiting confidence in it as a general monitoring metric.The limitation is tied to its origin in recent work.
- Information seeking: Proactive context information seeking remains difficult because the model must recognise what it does not know and formulate a query.The paper identifies this requirement as an inherently difficult research problem.
8. Call to Action
The paper calls for shifting from static self-play optimization to dynamic self-synthetic pipelines that sustain monotonic learnable information gain. It identifies three mechanisms for sustainable self-evolution: asymmetric co-evolution, capacity growth, and proactive information seeking.
- Sustainable self-evolution requires dynamic self-synthetic pipelines that guarantee monotonic learnable information gain.
- Asymmetric co-evolution exploits the computational gap between verifying and solving.
- Capacity Growth expands parameter and inference budgets in line with rising structural complexity.
- Proactive Information Seeking injects fresh context into the internal environment.
- Progress should be evaluated through bounded-observer metrics such as epiplexity, not solely downstream accuracy.
9. Conclusion
The paper attributes stagnation in self-evolving systems to failure to sustain increasing learnable information for bounded observers. It proposes an information-theoretic, system-level path from fragile self-play to continuous self-evolution.
- Self-evolving systems stagnate when they fail to sustain a monotonic increase in learnable information for bounded observers.
- Sustainable progress requires asymmetric co-evolution, dynamic capacity expansion, and proactive information seeking.
- The information-theoretic perspective provides system-level principles for transforming fragile self-play dynamics into robust, continuous self-evolution.
A. Experimental Setup
The experiments estimate epiplexity in small-scale self-play coding settings using Qwen models, prequential coding, and MDL-based selection. The setup varies model capacities, synthetic data directions, and self-play iterations.
- The study fine-tunes Qwen2.5 models from 0.5B to 14B parameters with LoRA on datasets generated by Qwen2.5 7B, Qwen2.5 14B, and Qwen3 4B.Ten percent of each dataset is held out for validation and bounded-entropy calculation.
- Training uses a maximum sequence length of 2048 tokens, learning rate 1e-4, and up to 20 epochs, with early stopping after five consecutive epochs without MDL improvement.Epiplexity is taken from the prequential code length at the minimum MDL.
- Self-play follows the Absolute Zero reinforcement-learning algorithm and trains a Qwen2.5 3B model, which is also used for epiplexity calculations.
- Epiplexity is computed from the prequential coded length at the intermediate training checkpoint achieving optimal MDL.The synthetic data is generated by the PROPOSER and used to train a SOLVER.
- The small-data computation assigns identical loss values within batches and aggregates per-token prequential code length and entropy into per-token MDL.
- For small datasets and multi-epoch training, losses are recomputed after full training to define a more accurate prequential coded length.
C. Reward and Epiplexity Across Iterations
Across nine self-play iterations, proposer and solver rewards increase while epiplexity fluctuates rather than following the same monotonic trend. The comparison shows why reward alone cannot reveal whether generative complexity remains learnable.
- Table A1 reports proposer reward, solver reward, and Epiplexity (×103 bits) across nine Experiment 2 iterations.
- Rewards gradually increase across induction, abduction, and deduction tasks, whereas Epiplexity fluctuates substantially.
- Reward improvement can coexist with unstable or degenerate generative complexity, so reward alone is insufficient to detect collapse.
- The epiplexity visualisations use MDL/Tok, prequential coded length/Tok, and entropy/Tok to identify the optimal MDL point.
- Epiplexity complements reward-seesaw analysis by distinguishing meaningful learnable structure from zero-sum reward redistribution.
D. Data Case
The data case presents three coding tasks distinguished by which elements of a program mapping are given and which must be inferred: abduction, deduction, and induction. It also provides task instructions and sample instances for each setting.
- Task samples: Figures A4, A5, and A6 provide sample instances for the abduction, deduction, and induction coding tasks, respectively.The accompanying data samples are labeled by task.
- Abduction: Abduction gives the model code and an output, requiring it to infer an input that would produce that output.This demands program analysis and inverse reasoning.
- Deduction: Deduction gives the model code and an input, requiring it to derive the resulting output without executing the program externally.This is described as the most common setting.
- Induction: Induction gives only input-output pairs, requiring the model to infer the underlying program and abstract the mapping's regularities.Multiple pairs are provided to make the program mapping unique, and this is identified as the most challenging setting.