PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Agent Skill Generation via Reinforcement Learning

Junhao Shen, Zhanqiu Zhang, Yiwen Guo, Hong Cheng

arXiv:2608.01678v1cs.LGcs.CL

TL;DR

Generating high-quality agent skills is difficult because skills lack a natural supervision signal for relevance or correctness. Skill-α uses reinforcement learning with progressive local editing and rollback rewards, improving average downstream success over the strongest baselines by 3.3 points on CL-Bench and 6.7 points on tau2-bench under GPT-4o.

  • Problem

    Skill generation lacks a natural supervision signal for skill relevance or correctness, despite the importance of automatically producing reusable skills from documents or experience.

  • Method

    Skill-α formulates skill generation as sequential local edits and trains each edit with rollback rewards from comparing original and edited skills on anchored downstream queries.

  • Results

    3.3 points on CL-Bench and 6.7 points on tau2-bench: Skill-α improves average downstream success over the strongest baselines under the GPT-4o worker in both document- and experience-to-skill settings.

  • Takeaways & Limitations

    Skill-α provides a unified learning framework that produces more effective skills across heterogeneous evidence sources and transfers gains across worker backbones.

  • Takeaways & Limitations

    Rollback comparisons provide noisy local signals, depend on benchmark-specific verifiers, and do not guarantee that repeated updates improve the final skill across its task family.

Abstract

from arXiv · show

Existing skill generation methods largely rely on heuristics or pipeline-style consolidation, which must be specially designed for different evidence sources. In contrast, learning-based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning-based skill generation remains challenging because skills lack a natural supervision signal based on relevance or correctness; their value can largely be determined only by whether they improve the behavior of the agent on downstream tasks. To address this challenge, we propose Skill-$α$, a reinforcement learning method for progressively generating high-quality agent skills. Specifically, we formulate skill generation as a sequential editing process that decomposes skill construction into individually evaluable edits, and introduce a novel rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query. Extensive experiments show that Skill-$α$ generates more effective skills than methods based on heuristics or pipelines in both document-to-skill and experience-to-skill settings. Under the main GPT-4o worker, Skill-$α$ improves average downstream success rates over the strongest skill-generation baseline by 3.3 points on CL-Bench and 6.7 points on tau2-bench. Further ablations validate the importance of rollback reward and progressive generation.

1 INTRODUCTION

Skill-α frames agent skill generation as progressive reinforcement learning over local edits, addressing the lack of direct supervision by grounding edit evaluation in downstream execution. It unifies document-to-skill and experience-to-skill settings and improves downstream success over strong baselines.

  • External skills guide agents’ task decomposition, tool invocation, and intermediate-result checking for multi-step reasoning and long-horizon planning.
  • Existing heuristic, prompting, and pipeline methods require separate designs for document-to-skill and experience-to-skill evidence sources.
  • Skill quality lacks direct relevance or correctness supervision and is instead assessed largely through effects on downstream task performance.
  • Skill-α formulates construction as sequential local editing, learning to add, revise, merge, remove, or preserve guidance based on execution feedback.
  • 3.3 points on CL-Bench and 6.7 points on tau2-bench are Skill-α’s average downstream success-rate improvements over the strongest baselines under the GPT-4o worker.Ablations validate rollback reward and progressive generation; analyses find robustness to evidence reordering and stronger dependence on evidence granularity.

2 RELATED WORK

Related work characterizes agent skills as reusable procedural modules and studies their evolution, selection, utilization, and internalization. Skill generation draws on diverse evidence sources, while reinforcement learning increasingly optimizes external modules such as tools, environments, and search use.

  • Agent Skills: Agent skills are reusable procedural modules that condition inference and reshape agent behavior on downstream tasks.Prior studies examine skill evolution, selection, utilization, and internalization.
  • Skill Generation and Acquisition: Skill generation and acquisition produce reusable modules from documents, task contexts, experience, and execution trajectories.Representative approaches include verifier-feedback memories such as ExpeL and Agent Workflow Memory, alongside prompt-based skill creation.
  • Reinforcement Learning for External Module Optimization: Reinforcement learning is used beyond policy optimization to manage external modules for complex task completion.Applications include improving tool invocation, environment interaction, and use of search results.

3 PRELIMINARIES

The paper models skill generation as progressively editing an initial skill from source evidence, with quality defined by its effect on a fixed worker’s behavior on held-out queries. Because the ideal teacher distribution is unavailable, the objective requires an indirect reward and is optimized using GRPO.

  • LLM Agent: A fixed worker policy produces trajectories conditioned on an externally inserted skill and task context, while the skill-editing policy—not the worker—is trained.The worker induces πψ(τ | z, q) from skill z and query q.
  • Group Relative Policy Optimization: GRPO optimizes the skill generator by sampling edit actions, assigning scalar rewards, computing group-relative advantages, and applying a clipped objective with reference-policy regularization.The objective uses the action-probability ratio ρi(ϕ), reference policy πref, and hyperparameters ϵ and β.
  • Skill Generation Formalization: Skill generation applies local edits progressively to evidence units, producing a distribution over final skills from an initial skill state.Evidence may include documents, task contexts, or execution traces.
  • Skill Generation Formalization: The ideal objective cannot be directly optimized because the conceptual high-quality teacher behavior distribution is unavailable, so skill generation relies on an indirect but related reward signal.The teacher distribution is denoted p⋆M(τ | q).
  • Skill Generation Formalization: Generated-skill value is measured by how the fixed worker’s behavior changes on a held-out target query from the same task family, typically distinct from the source query.The source evidence is induced by q′, while evaluation uses q, typically q′ ≠ q.

4 METHOD: SKILL-α

Skill-α formulates skill generation as progressive local editing: it sequentially processes evidence, applies structured edits, and trains edit decisions with rollback rewards based on downstream comparisons. The framework supports both document-to-skill and experience-to-skill generation while keeping the worker fixed during reinforcement learning.

  • 4.1 Progressive Skill Generation: Skill-α progressively transforms an initial skill z0 into a final skill zT through sequential local edits conditioned on the current state and evidence.At each step, an edit action At is applied to obtain the next state, and the formulation induces a distribution over final skills.
  • 4.1 Progressive Skill Generation: The progressive formulation unifies document-to-skill and experience-to-skill generation by reducing each evidence unit to a local editing decision.The decision is whether and how to edit the current skill so that the fixed worker performs better on held-out target queries.
  • 4.1 Progressive Skill Generation: The edit action space contains CREATE, UPDATE, MERGE, PRUNE, and NOOP, respectively adding, correcting, consolidating, removing, or preserving skill content.Each action edits a single skill artifact; NOOP is used when the evidence provides no useful new information.
  • 4.2 Rollback Reward: Rollback reward assigns edit-level credit by comparing worker outcomes under the original and edited skills on the same evidence-related anchored query.A benchmark-specific verifier evaluates both outcomes, avoiding credit assignment based merely on worker capability or query easiness.
  • 4.3 Training Implementation: During GRPO, the fixed GPT-4o worker evaluates the current skill and candidate edited skills, while benchmark-specific verifiers convert these scores into rollback rewards.CL-Bench uses a GPT-5.5 rubric judge with official task-specific rules; SpreadsheetBench and tau2-bench use direct benchmark-side environment feedback.

5 EXPERIMENTS

Experiments show that Skill-α performs strongly across document-to-skill and experience-to-skill settings, evidence sources, and worker backbones. Ablations indicate that rollback reward, structural editing, and progressive evidence handling are central to its gains.

  • Document-to-skill: Under GPT-4o, Skill-α raises Procedural Task Execution from 4.30 without skills to 9.68, exceeding Anthropic Skill-Creator, AutoSkill, and Ctx2Skill.The respective baseline scores are 5.38, 3.23, and 4.30.
  • Generalization Across Evidence Sources and Workers: Skill-α remains strong across evidence sources and workers, using one progressive editing framework where most baselines are designed for only document or experience evidence.Under Claude-Sonnet-4.5, it is best or tied-best on all four CL-Bench categories and achieves the highest CL-Bench average.
  • Experience-to-skill: Skill-α is best or tied-best across reported experience-to-skill metrics under GPT-4o, reaching a tau2-bench average of 55.83.It improves SpreadsheetBench from 18.00 to 27.50, Airline from 40.00 to 65.00, and Telecom from 12.50 to 22.50, while matching Retail’s best result of 80.00.
  • Ablation Performance: The full model gains +6.92, +12.00, and +11.66 over “SFT only” across the three benchmarks.Without rollback reward, performance stays close to “SFT only” at 3.68, 17.00, and 46.67; removing MERGE/PRUNE reduces the tau2-bench average to 39.17.
  • Evidence Order and Batch Size: Evidence order has a visible but nondeterminative effect, whereas batch size matters more, with 4 evidence units per step performing best.Both smaller and larger batches degrade performance, indicating that progressive generation depends on evidence granularity.

6 CONCLUSION AND FUTURE WORK

Skill-α frames skill generation as progressive editing of a single evolving artifact, training local edits with rollback reward on held-out evaluation anchors. It optimizes skills according to their effects on downstream worker behavior and produces more effective skills across three benchmarks.

  • Core approach: Skill-α formulates skill generation as progressive editing over a single evolving skill artifact.The framework trains local edit decisions using rollback reward on held-out evaluation anchors.
  • Core approach: Rollback reward evaluates skill edits by how they change downstream worker behavior rather than textual plausibility alone.This grounds edit optimization in downstream execution outcomes.
  • Empirical findings: Skill-α produces more effective skills on CL-Bench, SpreadsheetBench, and tau2-bench.The passage reports experiments across all three benchmarks without providing their individual numerical results.

A THEORETICAL ANALYSIS

The analysis establishes that rollback reward is a local pairwise ranking signal whose expectation equals the verifier’s probability of preferring an edited skill to the control. Under shared-branch assumptions, it preserves ideal-preference and binary-success ordering, while NOOP is characterized separately as a group-level fallback.

  • Pairwise interpretation: The comparison is consistent when control and edited branches share the query, worker, verifier, environment, and sampling protocol, with skill condition as the only systematic difference.The assumption also requires conditional mutual independence of branch rollouts.
  • Pairwise interpretation: Rollback reward is a Bernoulli sample whose mean equals the verifier pairwise win probability of an edited skill against the control skill.This interpretation applies to each valid non-NOOP candidate under the fixed local editing state.
  • Preference ordering: With candidate-independent verifier false-positive and false-negative rates satisfying η+ + η− < 1, expected rollback reward preserves ideal-preference ordering among non-NOOP candidates.Exact ideal preference is the special case η+ = η− = 0.
  • Binary success: Under an exact binary success verifier, expected rollback reward is proportional to edited-skill success probability when control success is below one, preserving candidate ordering.The expected reward is E[Ri] = (1 − β0)βi, with the common factor positive when β0 < 1.
  • NOOP behavior: NOOP reward depends on the sampled group of valid non-NOOP candidates and is a group-level fallback rather than an estimator of edit quality.If the group contains no valid non-NOOP candidates, Eq. 5 assigns NOOP reward one deterministically.

B IMPLEMENTATION DETAILS · B.1 SKILL-α SETUP

Skill-α uses a shared structured editing interface across document-to-skill and experience-to-skill settings, with sequential evidence processing, deterministic quality gates, and separate worker-verifier configurations. Its training begins with instruction-tuned Qwen3-8B and a synthesized warm-up corpus before reinforcement learning.

  • B.1 SKILL-α SETUP: The policy receives the current SKILL.md and serialized evidence, then emits a rationale and exactly one structured action.The action applier normalizes headings and applies the sampled edit without rewriting it into another action.
  • B.1 SKILL-α SETUP: Five actions support skill editing: CREATE, UPDATE, MERGE, PRUNE, and NOOP, which respectively create, modify, combine, remove, or preserve sections.CREATE supports multi-section creation; the other editing actions operate on existing sections, while NOOP preserves the artifact.
  • B.1 SKILL-α SETUP: The same edit policy is trained across evidence regimes, while benchmark-specific logic is limited to evidence serialization and reward evaluation.For CL-Bench, long contexts are processed as ordered natural segments, updating the current skill after each segment.
  • B.1 SKILL-α SETUP: Deterministic quality gates reject malformed, missing-target, duplicate-heading, empty-result, hidden-create-or-merge, and invalid-empty-skill NOOP edits before reward evaluation.Failing candidates receive reward 0 without benchmark evaluation, preventing exploitation through invalid or degenerate edits.
  • B.1 SKILL-α SETUP: Rollback reward always uses GPT-4o as the fixed worker during RL training, while final reporting evaluates GPT-4o and Claude-Sonnet-4.5 backbones.For CL-Bench, GPT-5.5 provides the verifier; SpreadsheetBench and tau2-bench use benchmark-side anchored execution feedback.

B.2 BASELINE SETUP

The baseline comparison standardizes shared evidence and preserves each method’s native construction and selection procedures. It includes a worker-only no-skill reference alongside document- and experience-based skill-generation baselines.

  • Common Settings: All skill-generation baselines receive the same source contexts or GPT-4o trajectory pool as Skill-α and construct one artifact per source context or experience group.Each method retains its native prompt package, artifact representation, update procedure, and selection rule; wrappers only serialize shared evidence and materialize native artifacts.
  • No Skill: NO SKILL applies no skill-generation procedure and runs the original benchmark interfaces, providing a worker-only reference for each backbone.It uses original CL-Bench messages or the official SpreadsheetBench and tau2-bench task interface without external guidance.
  • Prompt Baselines: Anthropic Skill-Creator and Progressive Prompt Skill use fixed GPT-4o prompts with temperature 0.1 and maximum output length of 2,600 tokens.Anthropic independently generates chunk-level skills and merges them, while Progressive Prompt Skill revises the current full skill using the next bounded evidence chunk.
  • Pipeline Baselines: AutoSkill, Ctx2Skill, and Trace2Skill are evaluated with their native document-ingestion, self-play, or trajectory-patch procedures on shared benchmark evidence.AutoSkill uses GPT-4o extraction; Ctx2Skill preserves multi-agent self-play and native selection; Trace2Skill orders successful executions first and constructs chunk-level Markdown patches.
  • Experience Baselines: ExpeL, AWM, SkillX, and SkillPro preserve their native experience-distillation, workflow-induction, hierarchical-library, or option-pool representations.These baselines use GPT-4o-based generation after converting shared trajectories to method-specific schemas, with successful executions ordered first and bounded trajectory inputs.

C EVALUATION SETUP · D PROMPTS

The evaluation spans document-to-skill and experience-to-skill benchmarks under matched source-evidence and held-out-task protocols, while the prompt interfaces define Skill-α’s progressive editing actions, reward evaluation, and baseline comparisons. Skill-α uses a shared evidence interface across training and generation, with benchmark-specific skill injection and evaluation procedures.

  • C EVALUATION SETUP: Experiments cover one document/context benchmark and two trajectory/experience benchmarks, with CL-Bench, SpreadsheetBench, and tau2-bench defining the evaluation scope.CL-Bench uses context-family splits, while SpreadsheetBench and tau2-bench use experience-benchmark constructions.
  • C EVALUATION SETUP: CL-Bench trains on Rule System Application and Procedural Task Execution, then evaluates generated skills on held-out in-domain and OOD contexts reused across tasks from each context.Domain Knowledge Reasoning and Empirical Discovery & Simulation form the OOD evaluation.
  • C EVALUATION SETUP: 400 tasks are used for SpreadsheetBench with a stratified 200/200 source/test split preserving the original cell-level and sheet-level task ratio.tau2-bench follows the official domain-wise split.
  • C EVALUATION SETUP: Each method receives identical source evidence and held-out tasks, with document methods restricted to contexts and experience methods restricted to trajectories, observations, and feedback.Generated skills are injected into the worker agent before evaluation under the shared protocol.
  • D.1 SKILL-α CORE PROMPTS: Skill-α’s shared interface supplies the current SKILL.md and execution evidence, then selects exactly one CREATE, UPDATE, MERGE, PRUNE, or NOOP edit.The interface is unchanged across document/context and trajectory/experience settings; only the serialized evidence payload differs.
  • D.2 REWARD AND EVALUATION: CL-Bench uses official task-specific binary rubrics with GPT-5.5 for final grading and rollback comparison, while SpreadsheetBench and tau2-bench use direct environment feedback on anchored executions.Experience-benchmark prompts prepend reusable skill guidance while preserving the original task prompts and environment harnesses.
  • D.3 BASELINES: Baseline prompts include Anthropic Skill-Creator and Progressive Prompt Skill, while other methods retain native prompts and representations for document- and experience-based skill generation.Progressive Prompt Skill shares progressive evidence exposure but lacks a learned editing policy and rollback reward.

E LIMITATIONS

Skill-α has limitations in its local edit-ranking signal and reward interface. A single rollback comparison is noisy and does not guarantee broader improvement, while transferring the method relies on benchmark-specific verifier design.

  • Appendix A provides only a local ranking result for candidate edits under the stated assumptions.This limitation concerns the scope of the reported ranking analysis.
  • A single rollback comparison supplies a noisy binary signal on one anchored query rather than an expected score difference.The signal therefore does not estimate how much an edit changes downstream performance.
  • The method does not guarantee that repeated local updates improve the final skill across its task family.The passage explicitly distinguishes local comparisons from guaranteed repeated-update gains.
  • The current reward interface still relies on benchmark-specific verifier design, limiting direct transfer to new environments.The supplied passage identifies verifier design as a dependency for applying the method beyond the benchmark setting.
Loading 2608.01678v1…