PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills

Siyuan Huang, Pengyu Cheng, Haotian Liu, Tao Chen, Yihao Liu, Jingwei Ni, Shijie Zhou, Ziyi Yang, Gangwei Jiang, Mengyu Zhou, Yu Cheng, Xiaoxi Jiang, Guanjun Jiang

arXiv:2607.22529v1cs.CL

TL;DR

Existing self-evolution methods trade task diversity against reliable verification. Skill Self-Play co-evolves task proposers, solvers, and an evolving skill library, yielding gains of up to +42.9 points on tool use and +12.0 points on logical reasoning.

  • Problem

    Existing self-play methods trade precise verification in narrow environments against broader task generation with less reliable validation.

  • Method

    Skill Self-Play co-evolves a proposer, solver, and dynamic controller that routes modular skills to generate and verify frontier-targeted tasks.

  • Results

    +42.9 points on tool use and +12.0 points on logical reasoning are the largest reported absolute gains from Skill-SP across evaluated backbones.

  • Takeaways & Limitations

    Skill-SP provides a structured self-evolution framework that expands task diversity while retaining executable boundaries for verification across tool-calling and logical-reasoning domains.

  • Takeaways & Limitations

    Skill-SP requires minimum foundational capability for novel task patterns, may need human demonstrations in complex domains, and relies on heuristics requiring empirical tuning.

Abstract

from arXiv · show

LLM training is shifting from manual design and annotation to interaction-driven self-evolution. However, existing self-evolutionary methods face a fundamental dilemma between task diversity and verification reliability: environment-bound methods obtain precise feedback but confine learning to narrow domains, while open-ended self-generation broadens the task space but lacks reliable verification, allowing misleading rewards to pollute the training loop. We identify agent skills as a powerful middle ground to reconcile this tension: each skill ensures deep, verifiable execution in a specific scenario, while dynamic routing across skills maintains open-ended task variety. Leveraging this insight, we introduce Skill Self-Play (Skill-SP), a co-evolutionary framework comprising a proposer, a solver, and a dynamic skill controller. Orchestrated via a reinforcement learning loop, these components co-evolve in a continuous self-play loop: the proposer generates challenging tasks conditioned on dynamically sampled skills; the solver explores candidate solutions to push its capability boundaries; and the skill controller collects execution feedback to update and expand the skill library. This interactive co-evolution effectively bridges the gap between structured verification and open-ended exploration. Empirical evaluations on tool-use and reasoning benchmarks demonstrate that Skill-SP, serving as a robust evolution engine, consistently pushes the performance ceiling of competent backbones while catalyzing striking turnarounds for initially misaligned models. Our code is available at https://github.com/Qwen-Applications/skill-self-play.

1. Introduction

Skill Self-Play addresses the tension between reliable verification and broad task diversity by routing modular skills through a self-play reinforcement-learning framework. Its Proposer, Solver, and Controller co-evolve skills and capabilities across verifiable tool-calling and logical-reasoning tasks.

  • Motivation: Self-evolution shifts LLM post-training from heavy annotation toward autonomous task generation, solution finding, outcome evaluation, and self-improvement.Self-play extends this paradigm through role-playing agents updated with multi-agent reinforcement learning.
  • Motivation: Environment-bound self-play provides precise validation but narrows training, whereas broader task generation risks weaker verification.The introduction frames this as a fundamental dilemma between task diversity and verification reliability.
  • Skill Self-Play: Skills are modular, reusable procedural-knowledge units that bundle task-specific instructions and supporting resources, enabling verifiable self-play beyond rigid environments.The skill concept is presented as a bridge between structured verification and broader task exploration.
  • Skill Self-Play: Skill-SP uses an evolving library of modular skill packages and orchestrates self-play through a Proposer, Solver, and Controller.The Proposer synthesizes skill-guided challenges, while dynamic routing and skill discovery support quality control and expansion of the task frontier.
  • Evaluation: Skill-SP is evaluated on verifiable tool-calling tasks from API-Bank and BFCL and logical-reasoning tasks from ZebraLogic across open-source LLM backbones.The paper presents these experiments as empirical validation of the framework’s ability to reconcile task diversity with verification reliability.

2. Related Work

Related work advances self-play and self-improvement through external verification, synthetic task generation, and reusable agent skills. These approaches respectively emphasize reliable feedback, broader task coverage, and skill construction from interaction and execution traces.

  • Self-play with external verifiers: Recent language-model self-play methods improve alignment, instruction following, and reasoning using self-generated data or verifiable rewards.Many self-evolving systems ground task generation in external environments to obtain reliable feedback.
  • Synthetic task generation and filtering: General self-improvement pipelines broaden task spaces by combining synthetic task generation with automated verification.Representative verification mechanisms include unit tests for coding, success signals in interactive web environments, and schema checks for tool use.
  • Skill interfaces for agents: Agent-skill research studies reusable procedural interfaces through retrieval, compression, and progressive disclosure, alongside automatic skill construction from interaction traces and execution feedback.The cited work covers both skill use and skill evolution.

3. Methodology

Skill Self-Play co-evolves a proposer, solver, and dynamic skill library to construct reliable, frontier-targeted curricula. It combines skill-conditioned generation, environment verification, reward-based task selection, and continual skill refinement, pruning, and induction.

  • Overall Framework: Skill-SP jointly optimizes proposer policy πpropose, solver policy πsolve, and an evolving skill library S in an iterative self-play curriculum loop.Skills route structural guidance to task generation, while proposer and solver policies co-evolve through training feedback.
  • Task Verification: Verifiable tasks pair a solver-visible prompt 𝒙 with a hidden machine-readable contract 𝒄, whose environment verification reward Rsolve(𝒙, 𝒚, 𝒄) lies in [0, 1].The contract can contain unit tests or reference answers and is evaluated exclusively by the environment.
  • Proposer Optimization: Skill-conditioned generation samples skills from S to provide procedural rules, hints, examples, executable validators, and usage statistics for proposer task synthesis.Sampling based on usage statistics balances exploiting high-yield skills with exploring under-tested skills.
  • Proposer Optimization: Validity verification requires schema compliance, skill-defined validator checks, and probe consistency, with K solver rollouts producing a unique majority answer matching the reference for skill-stream tasks.Exploration-stream tasks omit the skill-specific validation condition.
  • Solver Curriculum and Skill Evolution: The solver curriculum ranks valid candidates by proposer reward and selects top-scoring tasks from skill and exploration streams, while S co-evolves through skill refinement, pruning, and induction.Refinement diagnoses invalid-generation traces, pruning removes saturated skills yielding trivial tasks, and induction distills successful exploration into reusable structural priors.

4. Experiments

Experiments across tool-call prediction and logical reasoning show that Skill-SP improves all five backbones, including strong gains for misaligned models. Ablations and diagnostics attribute these gains to proactive, dynamic skill orchestration, co-evolutionary updates, and an expanding curriculum.

  • Experimental setup: Skill-SP evaluates five 3B–14B backbones on API-Bank, BFCL, and ZebraLogic, using identical checkpoints for the proposer and solver without an external teacher.The experiments cover schema adherence, tool selection, constraint satisfaction, and unique-solution deduction.
  • Tool-call prediction: 2.8 to 6.5 absolute points: Skill-SP improves all five backbones on diverse tool-calling scenarios, while Unguided SP produces smaller and inconsistent gains.Unguided SP can degrade specific subtasks, whereas Skill-SP reliably generalizes tool-use capabilities.
  • Logical reasoning: 12.0 points: Skill-SP’s overall grid-level gains on ZebraLogic reach this level for Ministral-3-14B and exceed 35 points on smaller puzzles.Skill-SP universally improves overall accuracy across all five backbones and task scales.
  • Ablations: 2.6 absolute points: Unguided SP drops overall performance, while Skill-only data degrades generalized API-Bank performance by over-specializing the curriculum.The findings support combining open-ended exploration with skill-routed data to preserve diversity and avoid mode collapse.
  • Ablations: 2.1-point overall drop: Freezing the proposer reduces performance, while freezing the feedback solver supplies outdated difficulty signals and breaks frontier tracking.The proposer must learn to leverage the evolving skill library through reinforcement learning.
  • Curriculum diagnostics: 0.57: Skill-routed tasks achieve a mean solver success rate closer to the learning frontier than Unguided SP at 0.70 and exploration at 0.75.Across five iterations, Skill-SP also induces roughly 20 new packages per round, updates existing skills from execution traces, retires obsolete skills, and grows accepted-pattern diversity to 46.

5. Conclusion

Skill Self-Play is presented as a co-evolutionary framework that addresses the tension between task diversity and verification reliability in LLM self-evolution. It uses a dynamically evolving modular skill library to steer high-fidelity curricula toward the solver’s current learning frontier, while active and effective skills grow steadily to expand the curriculum.

  • 5. Conclusion: Active and effective skills grow steadily, driving broad curriculum expansion.This summarizes the reported skill-utilization trend.
  • 5. Conclusion: Skill-SP addresses the fundamental tension between task diversity and verification reliability in LLM self-evolution.The framework is explicitly designed to resolve this tension.
  • 5. Conclusion: A dynamically evolving library of modular skills steers the proposer to synthesize high-fidelity curricula tailored to the solver’s current learning frontier.Skills function as proactive task-pattern interfaces within the co-evolutionary framework.

A. Iteration-wise Evaluation Trajectories

The section reports raw held-out performance trajectories across self-play iterations, starting from the initial solver at iteration 0. It evaluates tool-call and logical-reasoning performance across three solver backbones using distinct benchmark metrics.

  • Evaluation setup: Raw held-out performance is reported after each self-play iteration, with iteration 0 representing the initial solver.The trajectories cover the evolution process rather than only final performance.
  • Evaluation setup: Tool-call trajectories track the seven-subset overall score, API-Bank, and BFCL.These metrics are shown in the tool-call panels.
  • Evaluation setup: Logical-reasoning trajectories track ZebraLogic grid-level and cell-level accuracy.These metrics are shown in the logical-reasoning panels.
  • Backbone-wise trajectories: Figure 7 presents held-out tool-call and logical-reasoning trajectories for Qwen3-4B-Instruct.The figure places tool-call results on the left and logical-reasoning results on the right.
  • Backbone-wise trajectories: Figure 8 presents held-out tool-call and logical-reasoning trajectories for Qwen3-8B.The figure places tool-call results on the left and logical-reasoning results on the right.
  • Backbone-wise trajectories: Figure 9 presents held-out tool-call and logical-reasoning trajectories for Granite-4.1-3B.The figure places tool-call results on the left and logical-reasoning results on the right.

B. Details of Dynamic Skill Routing

Skill-SP dynamically routes each generation attempt by sampling skills from the skill library, balancing historically reliable skills with newly induced or under-tested ones. It estimates skill quality from multi-stage verification outcomes, then combines smoothed quality with decaying exploration to avoid premature routing collapse.

  • Dynamic skill sampling: Each proposer generation attempt dynamically samples a skill from the library to focus on structural priors that yield high-quality tasks.The routing mechanism balances exploiting historically reliable skills with exploring newly induced or under-tested skills.
  • Verification statistics: For every skill, tracking statistics record historical selection attempts and success counts across three stages of validity verification.The reported verification outcomes include structurally verified candidates and additional stages detailed in Section 3.2.
  • Smoothed quality estimation: Symmetric Beta smoothing with prior strength κ reduces the high variance of early sampling estimates for each verification outcome.Smoothed success rates are computed for verification, consistency, and frontier-difficulty outcomes.
  • Smoothed quality estimation: A composite skill quality score averages smoothed outcome rates, prioritizing structural validity while equally rewarding consistency and frontier difficulty.The score is defined as a weighted average of the smoothed rates.
  • Exploration-aware routing: Sampling weights add a decaying exploration bonus so under-tested skills remain sufficiently sampled and the routing distribution does not prematurely collapse.Clipping bounds [w_min, w_max] provide numerical stability, while β controls the bonus magnitude and τ controls its decay rate.

C. Training and Implementation Details … F. Computational Overhead

Skill-SP trains with verifier-grounded rewards, diversity controls, and structural filtering, starting from shared skill libraries that benefit substantially from online evolution. A complete five-iteration run takes slightly over one day, while explicit skill induction and refinement consume 6.5% of runtime.

  • C. Training and Implementation Details: Each training run uses eight NVIDIA A800 GPUs.These settings materially affect curriculum generation, solver optimization, and final benchmark scores.
  • C. Training and Implementation Details: Tool-call rewards weight exact function-and-argument matching at 0.9 and valid output formatting at 0.1.ExactCall requires a parsed call to match the hidden contract, while ValidFormat requires at least one parseable tool call.
  • C. Training and Implementation Details: Logical-reasoning rewards weight complete puzzle correctness at 0.9 and valid Zebra-grid structure at 0.1.Cell accuracy is logged diagnostically but is excluded from training rewards.
  • C. Training and Implementation Details: The proposer subtracts each prompt’s greedy-cluster proportion from its RL reward to encourage diversity and prevent batch-level mode collapse.Clusters use a Jaccard similarity threshold of 0.5, with penalty ρ(x_i) = |C_i|/B.
  • C. Training and Implementation Details: Generated tasks enter solver training only after structural parsing, domain-specific validation, and solver-consistency checks; Zebra puzzles must also be uniquely solvable.Tool-call records use package-level validators for skill streams and schema validity plus solver consistency for exploration streams.
  • D. Initial Skill Construction: The initial shared libraries contain 15 generic tool-call packages and 8 generic ZebraLogic packages, retained only after required-field parsing and verification checks.An LLM proposes packages from domain-level task formats and verifier interfaces using the Skill-SP representation.
  • D. Initial Skill Construction: Frozen skills reaches 64.4 overall versus 64.1 for Unguided SP and 66.7 for the full system, showing that online skill evolution supplies most gains.The ablation holds the initial library fixed while retaining skill-guided generation.
  • E. Full Pseudocode for Skill-SP: Algorithm 2 expands the high-level Skill-SP workflow into a single training loop.It provides the paper’s full pseudocode for the integrated procedure.

G. Analysis of Evaluation Benchmark · 25 end

The evaluation suite combines API-Bank and BFCL for complementary tool-call assessment with ZebraLogic for deterministically verifiable logical reasoning. Together, these benchmarks test executable action selection, tool-call generalization, and constraint-puzzle solving under structured verification.

  • G. Analysis of Evaluation Benchmark: API-Bank normalizes each instance into tool descriptions, dialogue history, and a hidden reference answer, requiring one exact tool invocation from the solver.Evaluation parses the predicted <tool_call> block and exact-matches the normalized reference.
  • G. Analysis of Evaluation Benchmark: BFCL complements API-Bank by testing tool-call generalization across distinct JavaScript, Python, Java, and live-simple benchmark categories.The categories stress function-schema conventions, type systems, and argument-normalization requirements.
  • G. Analysis of Evaluation Benchmark: Skill Self-Play is configured with a solver, an initial skill library, self-play iterations, a target-pool size, a curriculum ratio, and solver probes.The algorithm outputs a final solver π(T).
  • G. Analysis of Evaluation Benchmark: The self-play procedure updates proposal statistics from skill-conditioned attempts and refines skills using invalid attempts and accepted execution.These updates connect execution outcomes to ongoing skill-library refinement.
  • G. Analysis of Evaluation Benchmark: API-Bank and BFCL evaluate tool-call prediction from complementary benchmark sources, while ZebraLogic evaluates a separate logical-reasoning task family with deterministic verification.The suite covers selecting and grounding executable actions from dialogue context and solving deterministic constraint-satisfaction problems with unique answers.
  • 25 end: ZebraLogic evaluates logical reasoning on grid-structured constraint puzzles whose solutions assign attributes to houses.The evaluator checks parseable grids, schema agreement with the reference, and cell-level agreement with the hidden solution.

H. Limitations and Future Work · I. Case Studies of Induced Skills · I.1. Tool-Call Prediction

Skill-SP’s limitations concern bootstrapping novel task patterns and reliance on fixed heuristics, while its induced-skill case studies show reusable, validated tool-call transitions grounded in prior observations. Future work targets learnable orchestration, automated validator induction, and broader skill-library transfer.

  • H. Limitations and Future Work: Skill-SP requires minimum foundational capability to discover entirely novel task patterns and may benefit from human demonstrations in extremely complex domains.These demonstrations could jumpstart the evolving skill library.
  • H. Limitations and Future Work: The current framework relies on fixed heuristics, including a static skill-stream mixing ratio 𝛼.The passage also mentions additional predefined heuristics, but the supplied text truncates before specifying them.
  • H. Limitations and Future Work: Future work will replace fixed routing heuristics with learnable, dynamic curriculum schedulers to autonomously optimize data orchestration.This direction addresses how training data and skills are organized during evolution.
  • H. Limitations and Future Work: Future work will investigate fully automated co-induction of generative rules and executable validators directly from raw environment interactions.The goal is to automate both rule generation and validation from interaction data.
  • I. Case Studies of Induced Skills: All skill packages in the case studies were induced during Skill-SP runs initialized from Qwen3-4B-Instruct.This identifies the initialization model for the reported package examples.
  • I.1. Tool-Call Prediction: Each induced package records a reusable transition from an observed dialogue state to a valid next tool call, with generation constraints and a local validator.The examples illustrate library-level pattern induction.
  • I.1. Tool-Call Prediction: Complete listings contain routing metadata, curriculum state, proposer-visible construction fields, and a package-local validator, while solver procedures serve only as offline diagnostic oracles.The listings distinguish metadata and curriculum state, procedural construction resources, hints and validation, and package examples by color.
  • I.1. Tool-Call Prediction: Table 8 shows skill_038 grounding a top-rated restaurant in a reservation restaurant_id and skill_042 grounding a meeting interval in confirmed availability.Both packages were induced in iteration 2, use source=skill_induction metadata, and specialize different forms of observation grounding.

I.1.1. Complete induced package: skill_038 … I.2.1. Complete induced package: zebra_skill_022

The paper presents complete induced packages spanning restaurant booking, meeting scheduling, logical-reasoning constraints, and adjacency-based Zebra reasoning. These packages combine reusable execution rules with evidence-grounded validation and, where reported, deterministic structural checks.

  • I.1.1. Complete induced package: skill_038: skill_038 encodes booking a table at the top-rated restaurant after discovery, using specified guests, date, and time.It was added in iteration 2 through skill induction with specialization novelty.
  • I.1.1. Complete induced package: skill_038: 82 attempts were consistent, including 74 boundary-and-verified records and 8 too-easy records, with no too-hard or inconsistent records.These statistics characterize the restaurant-discovery-to-booking execution pattern.
  • I.1.2. Complete induced package: skill_042: skill_042 refines meeting scheduling by using confirmed availability observations rather than mirroring the user’s requested time range.The package schedules two or more participants within a specified window and extracts the topic from the request.
  • I.1.2. Complete induced package: skill_042: 1,993 attempts were consistent for skill_042, comprising 1,102 boundary-and-verified records and 891 too-easy records.The examples contributed filtered frontier records during routing, showing that the patterns function beyond descriptive retention.
  • I.2. Logical Reasoning: Logical-reasoning packages encode reusable constraint topologies through procedural construction rules and declarative compiler specifications.The fixed runtime admits an instance only when its schema, constraints, and unique solution pass deterministic checks.
  • I.2. Logical Reasoning: Meeting-scheduling validation requires ScheduleMeeting, nonempty participants and topic, ISO 8601 times, an ordered in-window slot, and no invented parameters.Times are normalized and copied from dialogue or confirmed observations, with all parameters required to satisfy the schema.
  • I.2.1. Complete induced package: zebra_skill_022: zebra_skill_022 adds a fixed-anchor adjacency bridge combining a fixed entity, cross-attribute next_to relation, and same_house component.The bridge resolves non-local positional dependencies through adjacency and shared-house structure rather than isolated left_of constraints.
  • I.2.1. Complete induced package: zebra_skill_022: 333 accepted and 4 rejected refinement records were independently saved for zebra_skill_022, while exported routing counters were zero.Every accepted record satisfied schema validity, constraint validity, uniqueness, and a capped solution count of one.

I.2.2. Complete induced package: zebra_skill_031

zebra_skill_031 is an iteration-2 induced chain–anchor–bridge skill that extends directed-chain reasoning with cross-attribute same_house and next_to bridges. Its refinement evidence is strongly validated, with 178 accepted and 2 rejected records, while routing counters remain unpopulated.

  • Library evolution: Induction evolves the library by adding reusable constraint topologies and compiler constraints that jointly structure directed chains, cross-attribute bridges, and uniqueness conditions.This logical-example evolution differs from the tool-call setting by exposing reusable structural constraints for subsequent generations.
  • Package identity and novelty: zebra_skill_031 adds a hybrid chain-solving structure in which a fixed anchor reinforces a directed adjacency chain through same_house and next_to bridges.The pattern propagates deductions through value-sharing and adjacency rather than position alone.
  • Package identity and novelty: The inducing trace uses the directly_left_of chain Elara–Gita–Leo, anchors Leo in House 3, and adds cross-attribute bridges absent from the source directed-chain skill.The added bridge types are same_house and next_to.
  • Validation evidence: 178 accepted and 2 rejected records comprise the saved refinement evidence, and every accepted record satisfies constraints_valid=1, schema_valid=1, puzzle_unique=1, and solution_count_capped=1.The exported stats.json contains no populated routing counters; all σ fields are zero.
  • Construction resource: The declarative compiler uses directly_left_of, same_house, and next_to, with seed programs for a length-3 anchored directed chain, a same_house chain, and a next_to component bridge.The runtime specification is version=1 with fixed_limit=1.
Loading 2607.22529v1…