Source-linked AI summary
SKILLER: Language-Level Reinforcement Learning for Reusable Skill Extraction in Small Language Models
Chenhao Dang, Siyuan Xiong, Conghui He, Weijia Li
TL;DR
Generating effective skills for compact models remains difficult and costly, so SKILLER uses natural-language reinforcement learning to optimize executor-specific textual skills and reports superior overall performance across both model scales.
Problem
Strong closed-source models make skill-based agent harnesses prohibitively expensive, while effective skills tailored to compact models remain difficult to generate.
Method
SKILLER treats textual skills as optimizable policies and uses a frontier-model actor-critic loop to refine them through structured natural-language feedback.
Results
SKILLER achieves superior overall performance across both Qwen3.5-9B and Qwen3.5-4B model scales on five diverse benchmarks.
Takeaways & Limitations
Executor-specific language-level policy optimization adapts procedural knowledge to compact models without updating their parameters.
Takeaways & Limitations
The optimization uses privileged evidence from a successful reference trajectory, although that trajectory is not provided to the compact model at runtime.
Abstract
from arXiv · showhide
Agent skills represent a standardized format for packaging procedural knowledge and domain expertise, serving within agent harness systems as an essential mechanism to continually constrain a language model's behavior space for repeatable, high-quality task execution. However, because strong closed-source models entail high inference costs, current popular agent harnesses, such as Codex and OpenClaw, remain prohibitively expensive when deploying these skills to accomplish real-world tasks. The rapid capability enhancement of open-source models deployable on consumer-grade GPUs presents a compelling opportunity to drastically reduce these costs by leveraging skill-based behavioral constraints. Nevertheless, automatically generating effective skills tailored specifically for such compact models remains a significant practical challenge. To address this, we propose SKILLER, a natural-language-driven reinforcement learning framework designed to automatically generate executor-specific skills for small models, which employs a strong model as the actor and critic, treats the small-model agent system as the environment, and propagates all reinforcement learning signals entirely via natural language. Extensive experimental evaluations across five relevant benchmarks using Qwen3.5-9B and Qwen3.5-4B demonstrate that SKILLER outperforms three open-source and one closed-source skill generation or evolution methods, achieving absolute gains ranging from 4.3 to 20.4 percentage points for the 9B model and 1.8 to 13.3 points for the 4B model, while remarkably matching the performance of strong closed-source models on single-skill tasks in SkillsBench. The project is available at https://github.com/DANG-ai/SKILLER.
Introduction
Agent skills package procedural knowledge and domain expertise, but reliance on strong closed-source models makes skill-based agent deployment expensive, while skills designed for frontier models do not reliably transfer to compact models. SKILLER addresses this mismatch by optimizing executor-specific textual skills through natural-language reinforcement learning and evaluates the approach across five benchmarks with Qwen3.5-9B and Qwen3.5-4B.
- Agent skills standardize procedural knowledge, tool-use conventions, and domain expertise for autonomous-agent harness systems.
- Strong closed-source models make reliable skill execution prohibitively expensive for real-world deployment at scale.
- Compact open-source models on consumer-grade GPUs can achieve high task success and substantially lower operating costs when constrained by appropriate skills.
- Skills built for frontier models often fail on compact LVLMs because their behavioral constraints, implicit reasoning, and error-recovery assumptions do not transfer.
- SKILLER treats textual skills as optimizable policies, using a frontier model as actor and critic while interacting with the target compact model’s agent loop.
- SKILLER is evaluated with Qwen3.5-9B and Qwen3.5-4B across four general-purpose benchmarks and the specialized EarthBench benchmark.
Related Work
Prior work frames agent skills as reusable procedural memory and studies their acquisition and evolution from diverse data sources. However, existing generation and evolution techniques primarily target frontier models, leaving compact-model constraints insufficiently addressed.
- Agent skills: Agent skills package procedural choices from tool-using interactions into reusable memory artifacts.Their evaluation across diverse benchmarks shows both utility and uneven transferability.
- Agent skills: Curated skills can improve execution success, whereas mismatched or partially relevant skills can degrade performance.
- Skill acquisition and evolution: Prior work acquires skills from repositories, large-scale networks, interaction histories, collective trajectories, and successful agent rollouts.Evolution frameworks also maintain executable libraries, store verbal feedback, and distill experience.
- Research gap: Existing skill generation and evolution methods are predominantly optimized for massive frontier models.The literature currently lacks methods specifically tailored to compact models’ constraints and execution paradigms.
Method
SKILLER optimizes executable natural-language skills for frozen compact models through verifier-grounded interaction, using frontier-model critics and actors instead of parameter updates. Its iterative loop diagnoses failures, applies localized edits, and accumulates reusable evidence to progressively improve task execution.
- Optimization framework: SKILLER treats the textual skill as its optimization variable, the compact-model agent loop as the environment, and the official verifier as the reward source.The compact model’s parameters remain unchanged during optimization.
- Environment and reward: Each rollout records an execution trajectory, scalar reward ri ∈[0, 1], and verifier diagnostics such as per-test outcomes and error messages.The reward is instantiated as task success or test pass rate.
- Critic: A frontier-model critic compares current and reference trajectories to identify causal divergences and convert verifier-grounded diagnoses into localized skill-editing instructions.It distinguishes procedural omissions, tool misuse, output-contract violations, and non-actionable infrastructure failures.
- Replay and iterative refinement: Replay memory stores failure signatures, critic summaries, and accepted edits with outcomes, while successive updates add finer constraints through iterative skill composition.The resulting progression moves from generic workflows toward grounded, executable, and self-correcting skills.
- Actor: A frontier-model actor applies bounded Insert, Replace, Create, and Delete operations, preserving effective content while adding constraints or deterministic task-local helpers.Reference evidence is distilled into runtime-input-dependent guidance rather than introduced as a direct runtime dependency.
Experiments and Analysis
SKILLER is evaluated across five benchmark families with compact Qwen3.5 models and a strong offline generator, outperforming baselines through executor-specific policy optimization. Analyses show gains on procedural tasks, transferability, iterative refinement, concise skill structure, and cost-performance trade-offs.
- Experimental setup: Evaluation spans five benchmark families, including single-skill SkillsBench tasks and execution-tested SWE-Skills-Bench instances covered by ten high-difficulty skills.The setup also includes SkillLearnBench, GAIA, and EarthBench.
- Experimental setup: SKILLER uses Qwen3.5-9B and Qwen3.5-4B executors in OpenCode, while GPT-5.4 acts only offline as actor and critic.Downstream evaluation tokens consumed by compact models are excluded from reported generation costs.
- Overall results: SKILLER outperforms open-source and closed-source baselines most clearly on complex procedural and strict tool-use benchmarks, including SWE-Skills-Bench and SkillsBench.Human-authored and generic Manus-generated skills can yield suboptimal gains or degrade performance relative to automated evolution baselines.
- Overall results: Qwen3.5-4B with SKILLER surpasses Qwen3.5-9B equipped with human-authored, automated, or Manus-generated skills on SWE-Skills-Bench.This result indicates executor-specific optimization can bridge substantial capacity gaps in structured real-world tasks.
- Generalization: On held-out GAIA and EarthBench data, SKILLER substantially outperforms all baselines on Qwen3.5-9B, suggesting reusable procedural rules rather than surface-level overfitting.The learned skills enforce systematic evidence gathering and strict output normalization.
- Optimization analysis: Across five iterations, both compact models continuously improve on SWE-Skills-Bench, while convergence speed depends on procedural complexity and later updates add finer execution constraints.Structural analysis finds SKILLER concise with low TF-IDF similarity, and cost analysis attributes its efficiency to targeted updates driven by execution feedback.
Conclusion
SKILLER is a natural-language-driven reinforcement learning framework that automatically generates executor-specific skills for small-scale LVLMs by addressing model mismatch through textual policy refinement.
- Conclusion: SKILLER treats the compact model’s agent loop as an interactive environment and refines textual policies using structured execution feedback without neural weight updates.Diagnostic signals are propagated entirely through structured text during iterative refinement.
Appendix A: State Analysis and Ablation
The ablation shows that task-instance and current-trajectory information ground effective policy updates, while reference trajectories and verifier diagnostics provide complementary refinement signals. This dependence is strongest on structured tasks with strict sequences, executable artifacts, and output contracts, whereas GAIA degrades less sharply.
- Experimental setup: The analysis evaluates each component of si = (x, τi, τ⋆, vi) by removing it while retaining the remaining optimization and evaluation settings.Table 5 reports average performance over three runs using each benchmark’s official metric.
- State-component ablation: Removing either the task instance x or current trajectory τi causes substantially greater damage than removing τ⋆ or vi, especially on SkillsBench and SWE-Skills-Bench.The task instance specifies executor requirements, while the current trajectory reveals how the skill shapes behavior.
- Benchmark dependence: The grounding pair matters most for benchmarks with strict tool sequences, executable artifacts, and output contracts.GAIA degrades less sharply under every ablation, retaining partial utility from broadly applicable search procedures when a state signal is absent.
- Refinement signals: Across all three benchmarks, removing the reference trajectory is more harmful than removing verifier diagnostics.Execution-process evidence offers a richer target for localized edits than outcome feedback alone.
Appendix B: Critic Prompt Analysis and Ablation
The critic prompt combines evaluation, trajectory comparison, earliest-error localization, and bounded skill generation, with ablations indicating that converting diagnosis into textual policy edits is decisive. Comparison and localization further improve causal repair, while prompt-based repair is less effective when failures depend on missing external evidence, as in some GAIA tasks.
- Critic operations: The critic performs four operations: evaluates execution, compares observed and reference trajectories, localizes the earliest causal error, and generates a bounded skill modification.The ablation removes each operation separately while retaining the remaining prompt instructions and experimental settings, using Qwen3.5-9B as executor.
- Critic operations: Removing Generation causes the dominant failure, showing that learning improves only when diagnosis becomes a bounded edit to the textual policy.Without generation, rollout evidence remains descriptive and cannot alter the behavior that produced the failure.
- Causal control: Comparison narrows the search to meaningful divergences, while earliest-error localization prevents downstream symptom patching beyond the verifier’s coarse outcome signal.The official verifier may partly compensate for missing evaluation, but it cannot identify which decision caused the outcome.
- Limitations: Prompt-based repair is less sensitive on GAIA because information-seeking failures may reflect missing external evidence rather than incorrect visible procedures.Structured skill and software tasks provide more repeatable causal traces, giving comparison and localization greater leverage; critics can reshape search policy but cannot supply missing facts.
Appendix C: Actor Prompt Analysis and Ablation
The actor prompt combines bounded editing, helper-script synthesis, preservation of effective content, and complete skill updates. Ablations identify helper synthesis as central for reliable execution, preservation as crucial for stability, and bounded editing as a consistent scope-control mechanism.
- Actor Operations: The actor prompt applies bounded edits, synthesizes task-local helper scripts, preserves effective skill content, and emits a complete skill update.The first three operations are ablated separately; removing the final operation would disable learning rather than isolate an operation.
- Actor Operations: Removing helper-script synthesis causes the strongest degradation on SkillsBench and SWE-Skills-Bench, highlighting executable abstraction as a bridge between feedback and reliable action.Task-local helpers move deterministic computation into reusable artifacts, reducing truncation, format drift, and inconsistent tool use by small executors.
- Actor Operations: Explicit preservation prevents the actor from replacing previously effective instructions when adapting to a critic signal derived from only one rollout.This stability constraint is particularly important on GAIA, where broadly useful search and verification routines must survive heterogeneous task-driven updates.
- Actor Operations: Bounded operations make critic feedback mechanically actionable by constraining plasticity through exact insertion, replacement, creation, and deletion primitives.Together, the three operations form an update policy in which helper synthesis adds executable competence, preservation retains it, and bounded editing mediates change.
Appendix D: Benchmark Task Selection and Data Splits
The appendix defines benchmark tasks and instances, then specifies skill-generation and zero-shot splits for SkillsBench, SWE-Skills-Bench, GAIA, and EarthBench. Stratified partitioning preserves benchmark composition while preventing held-out instances from updating skills.
- Benchmark Task Selection: 26 SkillsBench tasks were selected because each can be solved with a single skill.A task is a benchmark unit associated with a skill, while an instance is an individual evaluated sample.
- Benchmark Task Selection: 10 SWE-Skills-Bench tasks and 117 instances were retained when the original benchmark reported that adding a skill changed performance, including gains and declines.The subset targets settings in which skill effects can be observed.
- Data Splits: 83 GAIA instances formed the skill-generation split after stratifying by difficulty and randomly assigning half of each stratum, with the remainder reserved for zero-shot testing.GAIA is treated as one task, and each dataset sample as an instance.
- Data Splits: 124 EarthBench instances formed the skill-generation split using the same half-per-stratum procedure across three task categories, with remaining instances assigned to testing.EarthBench is treated as one task, and each dataset sample as an instance.
- Data Splits: Stratification preserves benchmark composition, and no held-out instance updates the skill, so zero-shot results assess transfer to unseen instances from the same benchmark.The partitions are generation and test sets created within each benchmark.
Appendix E: Method Implementation Details
SKILLER assigns each task its own skill bundle and experience buffer, initializes the bundle from the baseline skill and task-specific evidence, and uses fixed SkillsBench optimization settings.
- Task-Specific State: Each task receives an independent skill bundle and experience buffer, initialized by refining the baseline skill with the task instruction and reference trajectory.Unchanged local files are retained during initialization.
- Optimization Configuration: SkillsBench runs three optimization rounds, with each executor rollout limited to 30 tool steps.
- Optimization Configuration: Sampling temperatures are 0.2 for the executor and 0.3 for both the actor and critic.
Appendix F: Qualitative Skill Evolution
The appendix qualitatively traces three saved versions of the springboot-tdd skill, showing how feedback progressively converts open-ended procedures into budget-aware policies and artifact-consistency contracts while preserving the overall workflow. Across the revisions, seven execution phases remain unchanged, but their transition and completion criteria become more precise.
- Example Selection: The springboot-tdd example contains substantive, interpretable updates across three saved versions while preserving the same overall workflow.The versions were selected from recorded skills across all five benchmarks, and Figure 6 summarizes their semantic changes as steps.
- Step 1: Budget-Aware Control: The first transition turns a broad procedural guide into a budget-aware control policy with root anchoring, finite exploration, bounded evidence collection, and explicit file-checklist stopping conditions.These constraints address failures caused by limited interaction-budget allocation rather than missing domain knowledge.
- Step 2: Artifact Consistency: The second transition makes artifact consistency a repair precondition by requiring exact database targets, observed method signatures, cross-file dependencies, and valid tool arguments.The revision addresses failures where a locally plausible Java class conflicts with imports, repository methods, SQL profiles, or surrounding tool calls.
- Cross-Step Pattern: Across all three steps, the seven execution phases remain unchanged, while transition and completion criteria become more precise.New failure evidence becomes local policy constraints, while previously useful structure remains available to the executor.
Appendix G: SkillLearnBench Cost and Learning Dynamics
On SkillLearnBench, SKILLER’s effectiveness depends more on feedback-informed skill updates than generation volume, and its best checkpoint occurs before the final update. Across saved versions, the skill preserves its workflow while progressively adding constraints derived from execution failures.
- Cost and accuracy: Every reported SKILLER step exceeds the strongest baseline accuracy, despite AutoSkill using the most baseline-generation tokens without achieving the strongest baseline accuracy.SkillX performs better with a smaller token budget, separating generation volume from skill effectiveness.
- Learning dynamics: Step 2 is SKILLER’s best checkpoint, while later updates retain much of the gain without surpassing it.The nonmonotonic trajectory motivates snapshot and rollback because the final update need not be the most transferable policy.
- Cost and accuracy: The stage with the largest token usage does not yield the highest accuracy, indicating that update utility depends more on verifier-grounded feedback than raw generation volume.The five update stages consume similar token budgets, and the table reports stage-level rather than cumulative token usage.
- Skill evolution: Three consecutive saved skills preserve the same seven-phase workflow while converting observed execution failures into increasingly specific operating constraints.The evolving constraints target path discovery, edit planning, artifact consistency, and verification.