PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillGym: Training Skill-Use Agents with Automatic Verifiable Environment Generation

Renxi Wang, Mingshan Hee, Fajri Koto, Timothy Baldwin, Haonan Li

arXiv:2609.37539v1cs.AI

TL;DR

Public skills are widely used by LLM agents, but reliable training data and training methods for skill use remain underexplored. SkillGym turns curated skills into verified tasks and trajectories; finetuning improves skill-use performance across benchmarks and makes agents consult relevant skills more often.

  • Problem

    Public skills lack accompanying tasks, environments, and reliable success checks, while skill-use training approaches often derive skills from experience in a few fixed domains.

    Source: §1 · p. 1

  • Method

    SkillGym curates skills and uses a builder-reviewer pipeline to create skill-critical tasks across four reasoning structures, each with a reference solution and executable verifier.

    Source: §1 · p. 2

  • Results

    SFT improves six LLMs from three families, ranging from 2B to 122B parameters, across four skill-use benchmarks; Qwen3.5-9B outperforms Qwen3.5-397B-A17B on the SkillGym test set and SkillEval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Training raises the rate of consulting provided skills from 28% to 96%, with gains extending to underrepresented reasoning structures and skills held out from finetuning.

    Source: §1 · p. 2, §6 · p. 9

    The curated skills must support reproducible offline execution without network access, GPUs, or interactive input.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Skills equip LLM agents with professional knowledge and guidance to complete long-horizon and complex tasks. Although skills have been widely adopted in recent agent paradigms and harnesses, how to synthesize reliable training data and how to train agents for skill use remain underexplored. In this work, we propose SkillGym, an automatic pipeline to build verifiable environments, collect trajectories, and train skill-use agents. SkillGym first crawls a large volume of skills from the internet, then keeps those whose workflows can run reproducibly offline. A builder-reviewer pipeline is used to construct difficulty-controlled tasks, spanning four task types, each with a reference solution and an executable verifier. With this pipeline, we build 6.8k environments and collect 19k verified successful trajectories for supervised finetuning. Finetuning on these trajectories improves LLMs of different families and sizes, from 2B to 122B parameters across four skill-use benchmarks; Our Qwen3.5-9B SFT model outperforms the 397B untrained model on two of them. Further analysis shows that training teaches agents to invoke skills, raising the rate of reading the relevant skill from 28% to 96%, and that the gains hold across reasoning structures, extending to task types that form a minority of the training data and to skills held out from training

1 INTRODUCTION

SkillGym addresses the lack of task environments and reliable success checks for public skills by turning them into skill-critical training tasks. The paper reports that finetuning on its verified trajectories improves skill-use performance and increases agents’ consultation of relevant skills.

1 INTRODUCTION

  • Public skills lack accompanying tasks, environments, and success checks, while training approaches based on agents’ own experience remain confined to a few task domains.Tasks must make skill use consequential yet remain solvable without the skill, and their verifiers must recognize valid alternatives.Source: §1 · p. 1
  • SkillGym builds skill-critical tasks using four reasoning structures and a builder-reviewer system that creates and refines environments, reference solutions, and executable verifiers.Reviewers help identify ambiguous requirements, information leakage, and verifiers that are too restrictive or insufficient.Source: §1 · p. 2
  • 6 LLMs from three families improve on four skill-use benchmarks after SFT; Qwen3.5-9B outperforms Qwen3.5-397B-A17B on the SkillGym test set and SkillEval.The models range from 2B to 122B parameters.Source: §1 · p. 2
  • Training raises the rate of consulting provided skills from 28% to 96%, with gains across reasoning structures and transfer to structures that are a minority of training data.The gains also extend to tasks involving skills held out from finetuning.Source: §1 · p. 2

2 RELATED WORK

Related work spans training-free skill discovery, training-based skill-use learning, and task or environment synthesis. SkillGym instead starts from public skills to build verified training tasks and uses existing benchmarks to measure transfer.

Learning to Use Agent Skills

  • Training-based approaches update model weights using skill-related trajectories, whereas SkillGym keeps public skills external and trains general skill use that transfers to held-out skills.Prior approaches described here derive skills from limited environments or compile specific skills into model weights.Source: §2 · p. 2

Task and Environment Synthesis

  • Skill-driven synthesis starts from skills to create tasks: SkillGym combines explicit reasoning structures with verifier review, while SKT uses template-driven packages and its evaluation set concentrates on applying skill-provided rules.SkillGym’s tasks cover five annotated reasoning structures.Source: §2 · p. 3

Benchmarking Agent Skill Use

  • Existing skill-use benchmarks evaluate task completion or related capabilities and contain at most a few hundred tasks; SkillGym provides thousands of verifiable tasks for training and uses those benchmarks to measure transfer.The benchmarks assess varied demands, including retrieval, orchestration, procedural compliance, and boundary adherence.Source: §2 · p. 3, Figure 1 · p. 3

3 SKILLGYM

SkillGym turns curated public skills into executable training environments, using skill-critical tasks with reference solutions and verifiers. Its builder-reviewer pipeline constructs tasks and collects trajectories across diverse agent harnesses.

3 SKILLGYM

  • SkillGym’s three stages curate skills, construct skill-critical tasks, and collect trajectories for agent training.The pipeline starts from public skill packages and converts them into executable task environments.Source: §3 · p. 3

3.1 SKILL COLLECTION AND CURATION

  • The curated pool contains 11,897 skills selected for stable, modest-resource execution; eligible skills must be coherent, English-language, and avoid runtime network access, GPUs, destructive actions, and more than 300 files.A skill must also be complete, executable offline without interactive input, and clear enough to support meaningful workflow tasks.Source: §3.1 · pp. 3–4

3.2.1 DESIGN PRINCIPLE

  • Tasks must depend consequentially on skill-specific knowledge while remaining solvable through inspection and experimentation; each pairs a verifier with a reference solution.The verifier checks required outcomes while allowing valid alternative solutions.Source: §3.2.1 · p. 4

3.2.2 TASK PROFILES

  • Four task contracts cover dependent procedural steps, diagnosis of hidden causes, deliverables with competing measurable constraints, and DAG-ordered work with required joins and deliverables.The task profiles diversify how skills are applied.Source: §3.2.2 · pp. 4–5
  • 6,772 tasks from 3,494 skills span domains; successful teacher trajectories contain 19.1k tokens versus 15.6k for unsuccessful trajectories, with dashed lines marking the medians.These are the quantities and token lengths shown in Figure 2.Source: Figure 2 · p. 5
  • Each profile makes skill-specific knowledge consequential through realistic inputs, value-based verification, and a non-obvious step left unstated in the instruction.The hidden step may involve an edge case, failure cause, adversarial constraint, or join that otherwise produces an incorrect result.Source: §3.2.2 · p. 5

3.2.3 TWO-STAGE AGENTIC CONSTRUCTION

  • A builder-reviewer pipeline validates environments, assembles task packages, checks that verifiers reject the initial workspace and accept the reference-solved workspace, then reviews for leakage and overly strict or weak checks.The environment reviewer tests the built container; failed checks prompt builder revisions.Source: §3.2.3 · pp. 5–6

3.3 TRAJECTORY COLLECTION

  • Trajectories come from three open-weight LLMs and four harnesses, with varied prompts, tool names, schemas, and interaction formats to capture diverse behavior.The harnesses include Bash-based actions, dedicated file and command tools, and a JSON-based command protocol.Source: §3.3 · p. 6

4 EXPERIMENTS

SkillGym SFT improves results across four skill-use benchmarks, including on held-out skills. The experiments also compare performance with skills available versus unavailable and evaluate diverse model backbones under a common harness.

4.1 SETUP

  • Models are finetuned for two epochs on collected trajectories, and evaluation uses MiniSwe-Agent with bash as the only tool.The harness provides skill names and descriptions in the system prompt; agents must disclose detailed skill contents themselves.Source: §4.1 · pp. 6–7
  • Evaluation covers SkillGym, SkillEval, SkillsBench, and Skills-Use-Bench, reporting task success, mean normalized reward, and SU score as appropriate.The SkillGym test set separates unseen tasks with seen skills from unseen tasks with unseen skills.Source: §4.1 · p. 7

4.2 RESULTS

  • Training improves 22 of 24 comparisons, averaging +13.8 points on SkillGym test set, +9.7 on SkillEval, +9.7 on SkillsBench, and +41.2 in Skill-Use-Bench SU.The 9B model also outperforms Qwen3.5-397B-A17B on the SkillGym test set and SkillEval.Source: §4.2 · p. 7
  • SFT raises success from 40.0% to 57.0% on held-in skills and from 42.5% to 62.0% on held-out skills.Figure 3a reports results for Qwen3.5-9B, with skills available; the held-out split contains unseen skills.Source: §4.2 · p. 7, Figure 3 · p. 7
  • With skills available, success rises from 33.8% to 41.3% for the base model and from 43.0% to 59.5% for the SFT model.The SFT model’s success also rises by 9.2 points without skills, and its skill-access benefit is more than twice the base model’s.Source: §4.2 · p. 7, Figure 3 · p. 7

5 ANALYSIS

SkillGym SFT changes how agents use skills, while gains vary with task structure and skill content. The analysis tests data-review choices and whether tasks remain skill-critical for a strong model.

5.1 TRAINING IMPROVES MODEL’S BEHARIOR

  • Trigger rises from 28.0 to 96.0 after SFT, while Compliance rises from 32.5 to 49.0 and Boundary from 57.7 to 64.0.Trigger is the largest change; the base model opens the relevant skill in fewer than a third of tasks, compared with almost always for the trained model.Source: §5.1 · p. 8, Figure 4 · p. 8

5.2 TASK STRUCTURES

  • SFT improves all five reasoning structures by 16.9 to 22.1 points on SkillGym and 15.4 to 16.4 on SkillEval, with no structure-specific effect after controlling for difficulty.Rule application, the primary structure in only 12% of training tasks, improves by 13.9 points on SkillEval; on SkillsBench, reference-documentation tasks gain 28.0 points, while method-, tool-, and system-modification tasks do not improve.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3 DATA ABLATIONS

  • At matched token budgets, review-approved trajectories beat validated-only trajectories by 4.7 points on SkillGym, 6.7 on SkillEval, and 9.3 on Skill-Use-Bench completion.A mix of all four task profiles performs on par with procedural-only training on SkillEval and Skill-Use-Bench, and slightly lower on SkillsBench; these ablations are comparisons within each pair.Source: §5.3 · pp. 8–9, Table 2 · p. 9

5.4 ARE THE TASKS SKILL-CRITICAL?

  • Kimi-K3 success rises from 60.5% without skills to 68.8% with them, although 60.5% of tasks are solved without skills.The skill benefit is largest for procedural tasks (+16.0) and smallest for abductive (+4.0) and partial-order tasks (+3.0).Source: §5.4 · p. 9

6 CONCLUSION

SkillGym turns community-written skills into verifiable training tasks and trajectories, and SFT improves agents across skill-use benchmarks. The conclusion identifies method- and tool-centric tasks as an area of weaker gains, while the appendix describes how skills were annotated and selected.

6 CONCLUSION

  • SkillGym builds 6.8k skill-critical tasks across four reasoning structures and collects 19k successful trajectories; SFT improves LLMs across four benchmarks and gains extend to skills held out from training.The conclusion also reports that agents more often consult available skills and that review-approved tasks provide a stronger training signal than validated-only tasks.Source: §6 · p. 9

A SKILL ANNOTATION

  • Annotation labels cover 51,131 deduplicated skills; identity, runtime, and quality labels are then applied to 27,569, with 11,897 selected for task creation.An LLM annotator reads each SKILL.md and returns labels from closed vocabularies; out-of-vocabulary replies are rejected and re-requested.Source: A SKILL ANNOTATION · p. 12

B TASK PROFILE PROMPTS

Task profiles give builders and reviewers matched contracts for producing skill-critical tasks, while builder-reviewer stages construct and validate environments, tasks, and verifiers.

B TASK PROFILE PROMPTS

  • Each task profile is a builder prompt block paired with a reviewer addendum, and Table 4 describes skill-annotation fields, value shares, and selection rules.Table 4 notes that some fields allow multiple values per skill and that some are computed from repository metadata and file listings.Source: B TASK PROFILE PROMPTS · p. 12, Table 4 · p. 13
  • Shared profile rules require realistic inputs, independently recomputed outcomes, skill-sourced edge cases, and an unstated skill-specific step essential to solving the task.The verifier must check required outcomes rather than incidental solution form; the untouched workspace must fail and the reference solution must pass.Source: B TASK PROFILE PROMPTS · p. 12
  • Constraint-satisfaction tasks require at least three competing constraints, including a naive fix that measurably breaks another constraint.A checklist whose constraints can be satisfied independently is rejected; at least one satisfying choice hinges on unstated skill-specific knowledge.Source: B TASK PROFILE PROMPTS · p. 14
  • Partial-order tasks use a sampled 4–6-node dependency graph whose artifact-producing steps include real joins, skip edges, and fan-out.At least two intermediate artifacts are checked, and the task must defeat a serial, in-place strategy.Source: B TASK PROFILE PROMPTS · p. 14

C BUILDER AND REVIEWER PROMPTS

  • The environment builder creates a dependency image from the skill’s requirements, then the reviewer tests dependencies in use and reports build defects without patching them.The image provides dependencies only; skill files are mounted at run time rather than copied into the image.Source: C BUILDER AND REVIEWER PROMPTS · p. 14
  • The task builder packages an instruction, starting workspace, reference solution, verifier, and grading checklist around a concrete, automatically verifiable deliverable.The verifier checks outcomes rather than solver implementation and must pass when fact-carrying inputs change while reference outputs remain.Source: C BUILDER AND REVIEWER PROMPTS · p. 15
  • The task reviewer judges whether the verifier is the tightest test that accepts every correct solution, checking for both over-strict and too-weak checks.It can require revision for answer leaks, checks on unspecified choices, or outcomes that are not adequately tested.Source: C BUILDER AND REVIEWER PROMPTS · p. 15

D EXAMPLE TASKS

The examples illustrate four SkillGym task profiles with tasks that require skills for success; the procedural example is detailed, while the other profiles show distinct diagnosis, constraint, and planning cases.

Example tasks

  • All four example tasks are solved by Kimi-K3 with the skill but not without it, and by Qwen3.5-9B after—but not before—SkillGym SFT.The examples come from the SkillGym test set.Source: D EXAMPLE TASKS · p. 15

Procedural task

  • The procedural CO2 task chains derived quantities, time-series trajectories, and a report; its skill supplies formulas and conventions omitted from the instruction.An independent verifier recomputes expected values from inputs, including solubility, flow sign, and reversal time.Source: D EXAMPLE TASKS · p. 15

Abductive task

  • The abductive OASIS task asks agents to diagnose incorrect scores across 80 patients, where incorrect bin ordering mis-scores records with both low and high extremes.The verifier recomputes component and total scores for all 80 patients.Source: D EXAMPLE TASKS · p. 16

Constraint-satisfaction task

  • The brand-guidelines task requires eight constraints to hold simultaneously, including palette, font, contrast, formats, terminology, logo, and table styling.Palette and contrast requirements interact; the verifier recomputes contrast ratios and checks each constraint separately.Source: D EXAMPLE TASKS · p. 16

Partial-order task

  • The fourth example is labeled a partial-order finance task using the held-out edge-strategy-designer skill.Source: D EXAMPLE TASKS · p. 16

Procedural task

  • The procedural trading task joins concept drafts with calibrated reports, while a skip edge requires entry settings to survive for later export tickets.The obvious base risk values are wrong for some hypothesis types; the verifier recomputes calibrated values and checks each deliverable.Source: D EXAMPLE TASKS · p. 16

E ADDITIONAL REFERENCE MODELS

Table 5 presents reference-model evaluations with skills available under the protocol used for Table 1.

Reference models

F DATASET STATISTICS

The dataset-statistics materials describe the skill-curation funnel, task and trajectory summaries, and distributions across training and evaluation sets.

Dataset statistics

  • Tables 6–9 cover skill curation, selected-skill and task domains, task summaries, and training trajectories.Source: F DATASET STATISTICS · p. 16
  • Figure 6 compares reasoning-structure and skill-role shares across a stratified training sample and three evaluation sets.The samples are n=300 for training, n=400 for the SkillGym test set, n=100 for SkillEval, and n=87 for SkillsBench.Source: Figure 6 · p. 17
  • Figure 7 plots paired success changes from Qwen3.5-9B base to SkillGym SFT by reasoning structure, alongside each set’s overall gain.It shows task-bootstrap intervals and uses strict success for SkillEval and three-run reward for SkillsBench.Source: Figure 7 · p. 17
  • Table 6 tracks the skill-crawling funnel by source, including fetch failures, duplicates, screening, and cross-source overlap removal.Source: Table 6 · p. 18
  • Table 7 compares selected-skill and constructed-task domain distributions, with quotas making tasks more balanced than the skill pool.The quotas cap software engineering and exclude agents/meta and writing skills.Source: Table 7 · p. 18

G TASK STRUCTURE ANNOTATION

The section defines a three-facet rubric for labeling tasks and describes how an independent annotator applies it. It also specifies checks on annotation and summarizes the outcomes and comparisons reported for tasks and trajectories.

G TASK STRUCTURE ANNOTATION

  • The rubric labels reasoning structure, skill role, and action, while also recording leakage, verifier checks of undetermined conventions, evidence quotes, and annotator confidence.Reasoning structure includes five named labels plus a free-text “other” label, used once among 887 tasks.Source: G TASK STRUCTURE ANNOTATION · p. 18
  • The annotator reviews task instructions, workspace previews, skill documents, and verifier code, while reference solutions, design notes, and task metadata are withheld.Construction-profile identifiers are redacted; schema-invalid replies are returned with validation errors, and the annotated verifier-check count matches the evaluator specification for 99 of 100 tasks.Source: G TASK STRUCTURE ANNOTATION · pp. 18–19
  • Table 8 reports task counts by reasoning profile and review outcome, plus task-size statistics across 6,692 locally available packages.Partial-order tasks were constructed in separate runs and are excluded from the counts.Source: Table 8 · p. 19
  • Table 9 defines successful trajectories as those passing the task verifier and unsuccessful trajectories as graded failures, and reports trajectory-length statistics.Terminus-2 issues commands inside its JSON reply rather than as tool calls.Source: Table 9 · p. 19
  • Paired task outcomes compare one-run SkillGym success with three-run-average SkillEval strict success and SkillsBench reward, excluding permission-failure trials from the latter.The section reports 95% bootstrap intervals over tasks and describes regression of paired differences on structure labels, evaluation set, and difficulty.Source: G TASK STRUCTURE ANNOTATION · p. 19

H COST ESTIMATION

The cost estimate uses token counts and DeepSeek off-peak prices to estimate API expenses by stage. It puts the total at $5.7k–8.4k, or about $1 per released task, with task construction dominating.

H COST ESTIMATION

  • The estimate prices task construction at DeepSeek-V4-Pro and trajectory collection at DeepSeek-V4-Flash using cached-input, new-input, and output token counts.Table 11 gives the per-million-token prices and reports token counts in billions.Source: H COST ESTIMATION · p. 19, Table 11 · p. 20
  • SkillGym’s estimated API cost is $5.7k–8.4k, or about $1 per released task, excluding environment building, annotation, and evaluation.Task construction dominates the estimate, including failed and skipped attempts.Source: H COST ESTIMATION · p. 20
  • Reviewer cost assumes 12k input and 15k output tokens per call because reviewer transcripts were not stored.Token counts otherwise come from saved builder transcripts and a sample of 3,000 trajectories.Source: H COST ESTIMATION · p. 20
Loading 2609.37539v1…