Source-linked AI summary
SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation
Zelin Tan, Yiqun Zhang, Hao Li, Zhiyao Cui, Hejia Geng, Shao Zhang, Hangfan Zhang, Yang Chen, Xiaosong Wang, Lilong Wang, Zhenfei Yin, Shuyue Hu, Chen Zhang, Lei Bai
TL;DR
Models do not reliably identify, follow, coordinate, and execute externally supplied skills, leaving effective skill use underexplored. SKT synthesizes and verifies skill-grounded tasks and trajectories, and supervised fine-tuning on them consistently improves skill-use performance across models, benchmarks, and harnesses.
Problem
Models do not reliably identify, follow, coordinate, and execute relevant skills, while effective skill use remains underexplored.
Method
SKT selects skill configurations, synthesizes and verifies executable task packages, and collects validated trajectories for supervised fine-tuning.
Results
Across 16 comparisons, SKT improves mean score in every comparison, with absolute gains ranging from 3.20 to 18.91 points.
Takeaways & Limitations
Verified, composition-aware data synthesis is a practical route toward more reliable skill-using agents.
Abstract
from arXiv · showhide
Agent skills have become an important mechanism for equipping language-model agents with reusable procedural knowledge. However, providing skills alone does not guarantee that current models can effectively identify, apply, and coordinate them. To improve skill-use capabilities, we introduce SKT, a verified data synthesis pipeline that constructs skill-grounded tasks and executable trajectories from large collections of agent skills. SKT selects suitable single-skill and multi-skill configurations, synthesizes tasks through rule-based and agent-based verification with feedback-guided repair, and retains only successful trajectories that substantially use every required skill. Using 2,000 public skills, SKT produces 4,000 task packages and 27,164 verified trajectories. Based on the same pipeline and a disjoint test pool, we further construct SkillEval, a held-out executable benchmark for evaluating skill use. Experiments across diverse models, benchmarks, and agent harnesses show that supervised fine-tuning on SKT-generated trajectories consistently improves skill-use performance. Verification ablations, cross-harness evaluation, and scaling experiments further demonstrate that these gains depend on high-quality supervision, extend beyond a single agent interface, and increase with broader skill coverage. Together, these results establish verified data synthesis as an effective and scalable approach for skill-use training.
1. Introduction
Agent Skills extend LLM capabilities through reusable instructions, resources, and workflows, but models often struggle to identify and apply relevant skills effectively. SKT addresses this gap with a verified, controllable synthesis pipeline that produces skill-grounded training data and held-out evaluation benchmarks.
- Motivation: Agent Skills package instructions, metadata, and optional resources that extend LLM capabilities, support domain-specific tasks, and enable expert-knowledge reuse.They can also support composing multiple skills.
- Motivation: Effective skill utilization remains challenging because many LLMs cannot reliably identify relevant skills, follow their operational constraints, and coordinate their use.Skill availability alone does not guarantee effective utilization or improved task success.
- SKT framework: SKT synthesizes skill-grounded tasks and executable trajectories that are solvable, verifiable, and complexity-controllable, using curated public skills and verification-guided construction.The framework targets learning when and how to apply skills across diverse scenarios.
- Data and evaluation: 2,000 public skills yield 4,000 synthetic tasks and 27,164 verified execution trajectories, which supervise training for Qwen3.5-9B and Gemma 4 E4B-IT.Evaluation covers SkillsBench, MolBench-Bind, AgentSkillOS-bench, and SkillEval.
- Findings: SKT-trained models consistently improve skill-use performance across harnesses and benchmarks, with analyses attributing gains to high-quality supervision and showing generalization across interfaces.The studies include verification ablations, cross-harness evaluations, and scaling experiments.
2. Related Work
Related work develops external skills as evolving memories, temporary training scaffolds, or parameterized representations, while also advancing skill evaluation and agentic data synthesis. These approaches address skill lifecycle management, internalization, composition, benchmarking, and executable trajectory generation.
- Learning and evolving external skills: External-skill methods treat skills as persistent, editable memories that co-evolve with agents through creation, reuse, hierarchical distillation, and failure-driven updates.SAGE rewards skill creation and reuse; SkillRL updates a hierarchical SkillBank from validation failures.
- Skill internalization: Skill-internalization methods use external skills as temporary scaffolds, withdrawing them by helpfulness, contrasting paired rollouts, or distilling beneficial actions into skill-free policies.SKILL0 progressively withdraws skills, SkillC uses contrastive credit assignment, and SIRI mines and validates skills from successful rollouts.
- Parameterized skill representations: Parameterized-skill methods replace repeated skill-text injection with modular parameters, including separately loadable adapters and hypernetworks mapping textual skills to LoRA weights.LatentSkill and ParametricSkills investigate parameter-space composition, while ParametricSkills also trains on single- and multi-skill settings.
- Evaluating Agent Skills: Skill benchmarks evaluate execution-based utility, specificity, contextual compatibility, discovery, orchestration, and artifact production across domains and ecosystems.SkillsBench compares paired skill/no-skill execution; SWE-Skills-Bench evaluates software repositories; AgentSkillOS and MolClaw study ecosystem-scale organization and domain-specific hierarchies.
- Agentic data synthesis: Agentic data-synthesis methods generate compositional instructions, long-horizon computer-use tasks, and executable terminal trajectories with controlled complexity or error recovery.STEPS samples coherent skill combinations, AgentSynth controls complexity by subtask count, and TermiGen jointly synthesizes environments and trajectories.
3. Method
SKT is a three-stage pipeline that selects coherent skill configurations, synthesizes and verifies executable task packages, and retains faithful skill-use trajectories for supervised fine-tuning. Its verification combines deterministic checks, agent review, paired skill-dependence tests, difficulty control, repair, and trajectory validation.
- Skill configuration: SKT selects executable, objectively checkable skills and samples single- or multi-skill configurations, retaining multi-skill sets only when their roles form a coherent workflow.Experiments use configurations with k ∈ {1, 2, 3}.
- Task synthesis: TaskGen creates packages containing instructions, an isolated runtime, execution settings, an executable evaluator, and a reference solution, with an intended role for every selected skill.Packages must be objectively gradable.
- Task verification: Task packages pass deterministic and agent-based verification, including component, execution, semantic well-posedness, information-availability, and hidden-answer-leakage checks.The rule-based verifier also checks authoring-trace access to every skill and full evaluator credit from the reference solution.
- Task verification: Paired rollouts test whether skill access improves completion, while difficulty filtering and feedback-guided repair remove easy or defective tasks before they enter Dtask.Validation restarts at the rule-based gate after revision, and candidates exhausting the repair budget are discarded.
- Trajectory validation and training: Trajectory validators require full evaluator credit, normal termination, a well-formed trace, explicit skill access, and correct skill-guided decisions; accepted trajectories become harness-native masked SFT examples.Reasoning, tool-call, and response tokens are trained, while system, task, and tool-observation tokens remain masked from the loss.
4. Experiments
Across matched and cross-harness evaluations, supervised fine-tuning on verified SKT trajectories consistently improves skill-use performance, with gains depending on data quality and extending across agent interfaces. Mixed-harness training, broader skill coverage, and multi-skill settings further demonstrate the breadth of these improvements.
- Matched-harness evaluation: SKT improves mean scores in all 16 backbone–harness–benchmark comparisons, with gains ranging from 3.20 to 18.91 points.The smallest gain is Gemma 4 E4B-IT with OpenCode on SkillsBench, from 7.08 to 10.28; the largest is Qwen3.5-9B with DeepAgents on SkillEval, from 51.62 to 70.53.
- Skill dependence: SKT gains are larger when designated skills are provided, reaching 8.68–18.91 points versus 0.53–5.69 points when skills are withheld.The gap appears on both benchmarks and under both harnesses.
- Data quality: Unverified SFT lowers all four benchmark means, whereas SKT raises all four, producing gaps of 11.91–24.61 points.The comparison uses comparable raw-synthesis tasks and trajectories without verification or repair.
- Cross-harness transfer: Cross-harness SFT improves over Original in all four comparisons, yielding 4.35–9.86-point gains and retaining 49.1%–58.1% of matched-harness improvements.Matched-harness SFT remains stronger in every setting, indicating additional benefit from aligning trajectories with the target execution environment.
- Mixed-harness training: Mixed training improves over Original in all eight benchmark–harness comparisons, with gains ranging from 2.64 to 18.81 points, while remaining within 2.71 points of the matched Specialist.Mixed raises SkillEval from 55.24 to 74.05 under OpenCode and from 51.62 to 69.96 under DeepAgents, producing one checkpoint for either harness.
- Scaling and skill cardinality: SkillEval rises monotonically from 55.24 for Original to 72.48 at 2,000 training skills, while SKT improves normalized rewards for K=1, K=2, and K≥3 tasks.The corresponding gains are 7.40, 16.77, and 8.04 points, respectively, showing benefits beyond single-skill tasks.
5. Conclusion … A.3. Feedback and Stopping Policy
SKT combines verified, composition-aware synthesis with explicit control over task repair and trajectory retries, improving skill-use performance across models, harnesses, and domains. Its gains transfer across harnesses and increase with broader synthetic skill coverage.
- 5. Conclusion: SKT is a three-stage pipeline combining skill selection, feedback-guided task synthesis and repair, and trajectory synthesis with skill-use verification.It produces executable training examples for both single-skill and multi-skill workflows.
- 5. Conclusion: SKT consistently improves skill-use performance across diverse models, agent harnesses, and application domains.The learned behavior transfers across harnesses, while mixed-harness training enables one checkpoint to work effectively across different harnesses.
- 5. Conclusion: Performance further improves as synthetic skill coverage expands, establishing verified, composition-aware data synthesis as a practical route toward more reliable skill-using agents.The conclusion identifies broader skill coverage as a source of additional improvement.
- A. End-to-End SKT Procedure: The formalized SKT procedure constructs accepted task–configuration pairs before collecting verified execution trajectories, while retaining task, skill, teacher, and harness provenance.Inputs include externally represented Agent Skills C, requested skill cardinalities K, and configured teacher–harness pairs P.
- A.1. Inputs and Outputs: For each skill cardinality k, target Q_k accepted tasks, configuration bound B_select,k, repair budget B_task, and rollout budget B_traj govern pipeline capacity.B_traj bounds fresh rollouts for one task–teacher–harness pairing.
- A.2. Algorithm: QualifyTask applies rule-based, agent-based, and difficulty gates in order, while QualifyTrajectory combines deterministic outcome-and-trace checks with model-based skill-use checks.The agent-based gate includes semantic review and a paired skill-dependence test.
- A.3. Feedback and Stopping Policy: Feedback is used only for task repair: the first blocking task diagnosis guides TaskRepair, whereas rejected trajectories trigger fresh retries without verifier feedback.Finite repair and rollout budgets determine when each branch stops, and trajectory rejection does not modify the task package.
A.4. High-Level Experimental Configuration · B. Detailed Qualification Criteria · B.1. Verifier Inputs and Hidden Assets
The paper specifies SKT’s experimental configuration and makes task and trajectory qualification explicit through verifier interfaces, pass conditions, and recorded evidence. It also separates solver-visible inputs from hidden evaluator assets and defines what verifiers inspect.
- A.4. High-Level Experimental Configuration: Table 3 defines the high-level configuration needed to interpret Algorithm 1, while reproducibility details are reported separately.The separate details cover training, inference, hardware, serialization, and software.
- B. Detailed Qualification Criteria: Qualification criteria replace Algorithm 1’s abstract QualifyTask and QualifyTrajectory with explicit decision boundaries, verifier interfaces, pass conditions, and recorded evidence.The specification is independent of the authoring model and agent harness.
- B. Detailed Qualification Criteria: The construction algorithm takes skills, cardinalities, targets, teacher–harness pairs, budgets, and difficulty parameters, and produces task and trajectory datasets.Its stated outputs are tasks Dtask and trajectories Dtraj.
- B. Detailed Qualification Criteria: For each skill cardinality, SKT samples distinct suitable skills, checks composability for multi-skill sets, generates tasks, and accepts only those passing task qualification.Failed tasks may be repaired using verifier feedback before requalification.
- B.1. Verifier Inputs and Hidden Assets: A task package exposes instructions, setup artifacts, execution settings, and an output contract, while selected skills are mounted separately from hidden evaluator assets.Hidden assets include the evaluator specification, executable reference solution, and metadata recording each skill’s intended role.
- B.1. Verifier Inputs and Hidden Assets: Task verifiers inspect visible and hidden surfaces, whereas trajectory verifiers additionally inspect tool traces, output artifacts, and evaluator outcomes.In paired skill-dependence tests, solver access to the selected skills is the only changed condition.
B.2. Task-Side Qualification … B.5. Retry, Retention, and Recorded Evidence
SKT qualifies task packages through sequential rule-based, agent-based, and difficulty gates, then validates trajectories for complete, skill-dependent execution. Failed attempts are isolated and discarded, while only the first successful rollout and auditable verification evidence are retained.
- B.2. Task-Side Qualification: Accepted task packages must pass rule-based verification, agent-based verification, and difficulty control in sequence.The agent-based gate includes semantic task review and a paired skill-dependence test; every listed qualification condition is mandatory.
- B.3. Feedback-Guided Repair: Qualification stops at the first failed gate, and TaskRepair revises the complete package before restarting qualification from the rule-based gate.A task enters Dtask only after one package version passes all three gates.
- B.4. Trajectory-Side Qualification: Trajectory qualification combines a deterministic validator and an LLM validator across both supported harnesses.The deterministic validator requires full evaluator credit, normal and complete termination, a well-formed tool trace, and explicit access to required skills.
- B.4. Trajectory-Side Qualification: The deterministic validator blocks executions with failure, incomplete termination, malformed traces, or non-full evaluator outcomes.These four requirements are applied consistently despite harness-specific terminal signals.
- B.4. Trajectory-Side Qualification: The LLM validator accepts a skill only when it is consulted beforehand, changes a concrete decision or operation, and is applied correctly.Valid evidence can include skill-derived conventions, formulas, API patterns, thresholds, ordering constraints, resources, verification steps, or artifact-changing repairs.
- B.5. Retry, Retention, and Recorded Evidence: Rejected rollouts are discarded and retried in fresh sessions without prior traces, outcomes, failure reasons, or verifier feedback.The first rollout passing both validators is retained; exhausting the rollout budget yields no training example, and the retained example preserves the uninterrupted solver interaction.
- B.5. Retry, Retention, and Recorded Evidence: Verification records store gate verdicts, failure categories, task and skill identifiers, teacher and harness identifiers, evaluator outcomes, terminal status, validator verdicts, and skill-use evidence.These fields enable aggregate auditing without exposing hidden task assets, verifier prompts, or implementation source code.
C. Dataset Composition and Trajectory Statistics
The dataset comprises 4,000 accepted task packages built from 2,000 skills, with each package paired with four teachers and two execution harnesses.
- Dataset composition: 4,000 task packages form the accepted dataset pool used in the main experiments.These counts follow the same accepted task pool as the main experimental setup.
- Dataset composition: 2,000 skills selected from skills.sh underpin the task packages.
- Trajectory statistics: 4 teachers and 2 execution harnesses are paired with the task packages.
C.1. Task and Skill Composition
The accepted task pool is organized by designated skill cardinality, with each task paired across eight teacher–harness combinations before trajectory-side verification. A separate overlapping taxonomy describes represented functions without equating functional domains to skill-count tiers.
- Task composition: Each accepted task yields eight teacher–harness pairings, producing 32,000 candidate pairings before trajectory-side verification.The pairings cover one combination for each teacher–harness pairing.
- Functional taxonomy: The functional taxonomy is illustrative and overlapping, based on skill package names, descriptions, and optional metadata when available.Its categories are not exhaustive, and skills and tasks need not belong to mutually exclusive categories.
- Skill composition: The k = 1, 2, 3 tiers count designated skill artifacts, not functional domains.These compositionality statistics are kept separate from the overlapping functional taxonomy.
- Cross-family composition: A multi-skill task may combine skills within one functional family or across several families.For example, generating a statistical report can combine data analysis with document creation.
C.2. SkillEval Benchmark Composition … D.2. Optimization and Checkpoint Selection
The paper evaluates skill use on a held-out, executable SkillEval benchmark and describes trajectory measurement, serialization, optimization, and checkpoint-selection procedures. SkillEval is disjoint from training data, while SFT preserves native harness structures and evaluates the final one-epoch checkpoint.
- C.2. SkillEval Benchmark Composition: SkillEval contains 100 evaluation-only task packages and their trajectories, generated from a candidate pool separate from SFT data.The separation holds at both the task-package and trajectory levels.
- C.2. SkillEval Benchmark Composition: 30 tasks use one skill, 46 use two skills, and 24 use three skills, with distinct designated skill groups drawn from a held-out pool.The benchmark covers software development and debugging, data analysis and machine learning, security, and finance and business analysis.
- C.2. SkillEval Benchmark Composition: Evaluation runs each task in a fresh isolated session and workspace with its designated skills, reports mean normalized reward over 100 tasks, and repeats each model–harness condition four times.No-skill controls are included explicitly.
- C.3. Trajectory Measurements: Trajectory statistics are collection-level summaries whose N values match the main experimental setup, and interaction counts are reported as medians and 90th percentiles.Assistant messages count as turns, function invocations as tool calls, and parallel calls separately; these syntactic counts do not establish successful skill use or acceptance.
- D. Training and Evaluation Details: Training uses model-specific tokenizers and chat templates while retaining harness-native tool representations for Qwen3.5-9B and Gemma 4 E4B-IT.The implementation details are reported for both model families.
- D.1. Trajectory Serialization and Supervision: Retained rollouts are serialized end to end without mapping the two harnesses to a common tool vocabulary, supervising model-authored reasoning, tool calls, and responses.System and task messages plus tool observations are masked, records longer than 64k tokens are discarded, and history masking and packing are not used.
- D.2. Optimization and Checkpoint Selection: Full-parameter SFT uses LLaMA-Factory and DeepSpeed ZeRO-3; runs use fixed-seed shuffling, no optimizer-state resumption or validation-based selection, and evaluate the final checkpoint after one epoch.The common implementation settings are listed in Table 9.
D.3. Inference and Evaluation Protocol · D.4. Mixed-Harness and Scaling Variants
Evaluation uses harness-native agent loops with controlled decoding, isolated sessions, and benchmark-specific limits, while mixed-harness training balances verified trajectories and scaling varies skill coverage from 100 to 2,000 skills. Results are aggregated across four complete runs using benchmark-native 0–100 evaluators and arithmetic means with sample standard deviations.
- D.3. Inference and Evaluation Protocol: Models use OpenAI-compatible endpoints and harness-native loops: DeepAgents reads /skills/.../SKILL.md files, whereas OpenCode uses its native skill tool.Formal evaluation uses temperature 0.7; other decoding settings are fixed, while timeout watchdogs and interaction limits remain benchmark- and harness-specific.
- D.3. Inference and Evaluation Protocol: Original and SFT receive identical task assets and designated skills, except in explicit no-skill controls.Cross-harness evaluation changes the agent loop and skill interface, not the checkpoint.
- D.3. Inference and Evaluation Protocol: Every task starts in a fresh isolated session and workspace with no carried context or state.
- D.3. Inference and Evaluation Protocol: Four complete runs per model–harness condition produce aggregate scores reported as arithmetic means and sample standard deviations with n−1.Evaluation uses each benchmark’s native evaluator and the 0–100 metrics defined in the main setup, rather than task-level dispersion.
- D.4. Mixed-Harness and Scaling Variants: 24,000 verified records form the mixed-harness training set, sampled as 12,000 trajectories from each harness’s final pool.
- D.4. Mixed-Harness and Scaling Variants: Mixed-harness training preserves each harness’s native message and tool-call representation without injecting a harness identifier into training messages.The checkpoint otherwise uses the common SFT settings in Table 9.
- D.4. Mixed-Harness and Scaling Variants: Scaling trains Qwen3.5-9B on verified OpenCode trajectories from 100, 500, 1,000, or 2,000 skills and evaluates on SkillEval with designated skills.Zero budget denotes the off-the-shelf Original, not a separate SFT run.