Skill-R1: Agent Skill Evolution via Reinforcement Learning

1 Introduction

• Existing approaches rely on fixed skills, prompt edits, or task-model fine-tuning, motivating reward-aligned recurrent skill optimization.The paper frames useful skill evolution as improving current rollouts and turning observed outcomes into a better next-round skill.Source: §1 · p. 2
• Skill-R1 trains a lightweight skill generator to revise skills from task context, prior rollouts, and verified outcomes while keeping the task LLM frozen.This preserves compatibility with open- and closed-source task models and can be more efficient than updating the task model itself.Source: §1 · p. 2
• Each generation uses the current skill to induce rollouts, verifies their outcomes, and feeds trials, errors, and rewards into the next revision.The process repeats on the same instance, allowing directional improvement rather than selecting among isolated one-shot attempts.Source: §1 · p. 2
• The bi-level group-relative objective combines within-generation rollout comparisons with across-generation measures of whether revisions improve verified performance.The two signals reward both strong rollouts and beneficial skill revisions.Source: §1 · p. 2
• Skill-R1 is evaluated on agent tasks with verifiable rewards and outperforms no-skill baselines and standard GRPO.The paper reports consistent gains across reasoning and tool-use benchmarks and a performance gap between skill generations.Source: §1 · p. 2

2.1 Agent Skills

• An agent skill is a reusable external procedural artifact that conditions a frozen task model's execution on an input instance.It may encode natural-language instructions, workflows, tool guidance, or other procedural constraints.Source: §2.1 · p. 3
• For an instance and skill, the frozen task model induces rollouts; recurrent evolution tracks the generation's skill and instance-specific history of prior rollouts and verified outcomes.The notation names the generation skill as s_g and the history available up to generation g as H_g.Source: §2.1 · p. 3

2.2 Group-Relative Policy Optimization

• GRPO scores sampled rollouts with a verifier and uses group-relative advantages to promote rollouts that outperform peers under the same input context.The review introduces the group-based comparison used as a basis for the paper's optimization objective.Source: §2.2 · p. 3

3 Skill-R1: Agent Skill Evolution via Reinforcement Learning

• Skill-R1 separates task execution by a frozen task LLM from skill improvement by a separate editor policy using verified outcomes.The overview depicts recurrent skill generation, rollout, verification, and history storage, with the skill generator updated while the task model stays frozen.Source: §3 · p. 3, Figure 1 · p. 4

3.1 Problem Setup

• For each instance, the goal is to revise skills so later rollout populations achieve higher verifier reward.Each generated skill is evaluated through its induced rollout population, and verifier outcomes are added to the instance-specific history for subsequent revisions.Source: §3.1 · pp. 3–4

3.2 Bi-Level Group-relative Advantages

• The bi-level advantages combine same-skill rollout comparisons with cross-generation progress, which within-generation comparisons alone cannot capture.The inter-generation term measures changes in mean reward; λ controls its contribution, with λ=0 yielding pure within-generation GRPO.Source: §3.2 · p. 5

3.3 GRPO Objective for Skill Evolution

• The clipped GRPO objective aggregates bi-level advantages across generations, training the skill generator to produce improving skill trajectories without updating the task LLM.The task model remains frozen, and no gradients through it are required.Source: §3.3 · pp. 5–6

4.1 Experimental Setup

• The evaluation uses GAIA’s 165 heterogeneous-source tasks and WebWalker’s multi-hop web-navigation tasks, with No Skills and Vanilla GRPO as baselines.Skills are initialized by distilling traces from ten seed tasks; experiments distinguish a frozen-editor inference setup from online GRPO training of the editor.Source: §4.1 · p. 6

4.2 Results for GAIA

• 41.8% GAIA accuracy is achieved by GRPO-trained Skill-R1, a +12.1 point gain over Vanilla GRPO; Level 3 accuracy is 38.5%, versus 15.4% for Vanilla GRPO and 0.0% without skills.The inference-only setup achieves 30.9% overall GAIA accuracy.Source: Table 1 · p. 6, §4.2 · p. 7

4.3 Results for WebWalker

• 26.0% WebWalker accuracy for the GRPO-trained editor is a +4.0 percentage point gain over Vanilla GRPO; inference-only Skill-R1 reaches 19.0%.The trained editor reaches 29.4% in medium tasks and 28.6% in multi-source tasks.Source: §4.3 · p. 7, Table 2 · p. 7, Table 3 · p. 7

5.1 Reward and Accuracy Progression Across Generations

• Generation 5 reaches 0.44 running mean reward and 50.0% accuracy, versus Generation 1’s 0.06 reward and 9.3% accuracy; Generation 4 reaches 0.37 reward and 46.3% accuracy.The largest gains occur between Generations 1 and 3, with later rounds increasingly refining the skill set.Source: §5.1 · pp. 7–8, Figure 2 · p. 8
• Reward gains exceed accuracy gains from Generation 4 to 5, indicating more consistent successful rollouts on tasks that are already solvable.Accuracy records whether any rollout solves a task, whereas reward reflects successful-rollout consistency.Source: §5.1 · pp. 7–8

5.2.1 Trained v/s Inference Skill Editor

• The GRPO-trained editor reaches 41.8% on GAIA and 26.0% on WebWalker, while inference-only editing reaches 30.9% and 19.0%, respectively.In the pdb case study, the trained editor preserves executable structure and adds safeguards, whereas inference-only editing can discard essential components and produce brittle task-specific artifacts.Source: §5.2.1 · pp. 8–9

5.2.2 Editing Trajectory Across Generations

• Skill evolution follows a prune-then-specialize trajectory: early generations compress broad references into executable routines, while later edits add constraints and guardrails for recurring failures.Most gains arrive early; subsequent rounds primarily improve consistency and reliability.Source: §5.2.2 · p. 9

6.1 Agent Skill in Language Models

• Agent-skill research spans skill libraries, autonomous discovery and distillation, verifiable programmatic skills, and system-level coordination; newer work explores iterative skill evolution, often through heuristic refinement or implicit feedback.Examples include SkillWeaver (Zheng et al., 2025), ASI (Wang et al., 2025d), MemSkill (Zhang et al., 2026), and Memento-Skills (Zhou et al., 2025).Source: §6.1 · p. 9

6.2 Reinforcement Learning for Language Models

• Language-model RL ranges from PPO and GRPO to agent skill learning, but most existing methods optimize the task model or use a single-generation setting.SkillRL (Xia et al., 2026) jointly learns hierarchical skills and policies through recursive reinforcement learning.Source: §6.2 · p. 9

• Skill-R1 consistently outperforms no-skill baselines and standard GRPO across benchmarks, with the largest gains on complex, multi-step tasks.Source: §7 · p. 10
• Early generations drive most capability improvements, while later iterations enhance reliability and consistency.Source: §7 · p. 10
• Skill-R1 decouples execution from skill refinement and uses a bi-level optimization objective for iterative, reward-driven improvement over multiple generations without modifying the task model.Source: §7 · p. 10

• The appendix compares the original PDB skill with revisions from a GRPO-trained Qwen3-4B editor after three generations and an untrained base Qwen3-4B model in inference mode.Source: §A.1 · p. 13

• The original skill covers BioPython’s full API, with the appendix showing its header and Quick Start as examples of its breadth.The remaining sections cover dependencies, PDB format basics, Python libraries, manual parsing, common tasks, and a quick-reference table.Source: §A.1.1 · p. 13
• The skill directs users to apply it to reading, parsing, analyzing, or manipulating PDB files, including extracting coordinates and identifying residues, chains, and ligands.Source: §A.1.1 · p. 13
• The Quick Start illustrates BioPython parsing through the Structure-to-Atom hierarchy, while the format overview identifies ATOM, HETATM, SEQRES, and REMARK records.Source: §A.1.1 · p. 14

A.1.2 GRPO-Trained Editor: Edited pdb Skill

• The GRPO-trained skill retains the original description and Quick Start structure while adding error prevention and PDB-only scope guidance.Source: §A.1.2 · pp. 14–15
• The GRPO-trained editor adds checks for missing chains and residues and includes guidance for missing files or BioPython, while preserving the original reference.Its skill also states that it is for PDB files, not audio recordings.Source: §A.1.2 · pp. 14–15

A.1.3 Base Inference Editor: Edited pdb Skill

• The base inference editor replaces the skill with content about a GAIA task UUID and an unrelated audio homework question, leaving no executable code.Source: §A.1.3 · p. 15
• The GRPO-trained editor adds code for real failure modes while retaining the reference; the base inference editor removes code and copies benchmark-question content, leaving it useless for unseen PDB tasks.Source: §A.1.3 · p. 15