Source-linked AI summary
SPT: Skills as Pre-Training Data for Agentic Language Models
Yufei Sun, Yudong Li, Yiming Cheng
TL;DR
Agentic models lack broad, naturally collected tool-use processes, while producing verified trajectories is expensive. This paper introduces SPT, which mid-trains on public multi-file skill packages with reference-aware assembly and optional general-data mixing; across scales and post-training settings, it consistently improves agentic performance while largely preserving general performance.
Problem
Complete tool-use processes are scarce in natural corpora, and producing valid trajectories requires task environments, execution, and verification.
Method
SPT applies causal language modeling to a cleaned SkillCorpus of public multi-file skill packages, using Reference Insert and optional general-data mixtures before post-training.
Results
SPT consistently outperforms direct SFT and general- or trajectory-data mid-training on agentic performance across evaluated scales and configurations, while largely preserving general performance.
Takeaways & Limitations
Skill packages provide reusable workflow knowledge that can serve as pre-training data for agentic language models beyond inference-time context.
Takeaways & Limitations
The corpus reflects a single ClawHub community and underrepresents some languages, tool ecosystems, and specialized agentic domains.
Abstract
from arXiv · showhide
Agentic (tool-using) language models are mainly trained on tool-call traces and agent trajectories during post-training. These data provide direct behavioral supervision, but producing them requires task environments, execution, and verification, making broad tool and task coverage expensive. Publicly available skills offer another source of training data: they encode reusable tool semantics and workflows but are typically used only as inference-time context. We introduce Skill Pre-Training (SPT), a mid-training method that applies causal language modeling to SkillCorpus, a collection of public multi-file skill packages, optionally mixed with general data. To preserve relations among files within each package, we also introduce Reference Insert, a reference-aware assembly strategy that places supporting files near their mentions in the primary instruction. Experiments across multiple model scales and post-training recipes show that SPT consistently improves agentic performance over mid-training on general or trajectory data, while largely preserving general performance. Data mixture experiments show additional benefits from combining skill data with general annealing corpora. These results indicate that skill packages are a valuable data source for pre-training agentic language models.
Introduction
Agentic models need tool-use knowledge and multi-step interaction data, but complete trajectories are costly to produce. SPT uses public, reusable skill packages for mid-training and reports consistent agentic gains with little change in general performance.
- Tool-use capabilities require models to understand tool functions and perform multi-step interactions.
- Public skills encode reusable task instructions, references, scripts, templates, and configurations rather than one particular execution.
- 9.6× growth in publicly available npm skills from February to June 2026 indicates a rapidly expanding data source.The reported count increased from 170,226 to 1,640,440 skills.
- SPT applies causal language modeling to SkillCorpus during mid-training before downstream behavior-oriented post-training.SkillCorpus contains 38,040 cleaned and decontaminated ClawHub packages, with Reference Insert preserving instruction-resource relations.
- Across model scales and post-training configurations, SPT improves agentic performance while largely preserving general performance.Mixtures with general data and downstream reinforcement learning provide additional or retained benefits.
Related Work
Prior work mainly trains tool use through calls, trajectories, and post-training, while related skill systems focus on organizing reusable assets. SPT instead treats multi-file skill packages as mid-training data and evaluates their effects on distinct agentic behaviors.
- Mid-training adapts pretrained checkpoints on curated corpora, whereas post-training teaches instruction following and response behavior.
- Prior language-modeling corpora expose models primarily to tool calls, verified examples, search traces, or realized interaction trajectories.
- SPT changes the training unit to reusable multi-file workflow specifications containing tool semantics, procedural constraints, and resource relations.
- SkillNet and SkillCenter emphasize skill construction, organization, acquisition, filtering, and agent-side reuse rather than mid-training effects.
- Existing benchmarks diagnose tool awareness, selection, argument validity, chaining, planning, feedback consistency, web interaction, and long-horizon behavior.SPT uses these benchmarks to measure how mid-training corpora affect distinct agentic components.
Skill Pre-Training
SPT mid-trains models on cleaned multi-file skill packages, optionally mixed with general data, while preserving relationships among instructions and supporting resources. Reference Insert localizes referenced files, and experiments compare alternative organizations and mixture ratios.
- Skill Pre-Training: SPT exposes base models to human-written tool descriptions and reusable workflows before SFT teaches response and tool-call formats.
- SkillCorpus Construction: Skill packages combine a primary instruction file with optional references, scripts, templates, configurations, and other text resources.
- SkillCorpus Construction: 38,040 packages and 218,277 files remain after quality screening, decontamination, cleaning, and preservation of package boundaries.The corpus averages 5.74 files per package; about 2% of collected packages are removed during quality screening.
- Multi-File Skill Assembly: Reference Insert places each unambiguous supporting-file reference immediately after its first mention, then appends remaining files with separators.This keeps referenced instructions and resources local without changing the package’s underlying files and text.
- Multi-File Skill Assembly: Serialized packages form fixed-length training blocks while retaining file and package boundaries, and alternative organizations test the value of reference-aware serialization.
- Skill and General Data Mixture: α denotes the skill-data fraction in a fixed-token-budget mixture with general data, with α = 1 as pure-skill training and α = 0 as a control.No new training objective is introduced; the model continues causal language modeling on the mixed corpus.
- Skill and General Data Mixture: Mixture experiments evaluate whether skill data should replace or complement general annealing data across different agentic and general-performance tradeoffs.
Experiments
Across three model scales and multiple post-training settings, SPT improves agentic performance over direct SFT and general- or trajectory-data mid-training while largely preserving general performance. Skill data benefits from reference-aware assembly and from mixing with general data, and its advantage persists after reinforcement learning.
- Main results: Across all six backbone and SFT settings, SkillCorpus achieves the best agentic performance and ranks first in 23 of 24 individual comparisons.It outperforms both the general corpus Dolmino and the agent-trajectory corpus AgentBank.
- Main results: SPT improves the four-benchmark agentic score over direct SFT by 9.11–24.96, while the six-benchmark general score changes by only −0.85 to +0.51.The pattern holds across the evaluated backbones and post-training recipes.
- Corpus comparisons: SkillCorpus outperforms AgentBank by 4.07–9.91 under the same training-data budget.Skills explicitly state tool applicability, constraints, and reusable procedures, whereas trajectories record one execution’s actions and observations.
- Post-training recipes: SPT’s gain over direct SFT is larger after xLAM-FC than after Tulu 3 at every backbone size.The gains are 11.55 versus 9.11 at 1.6B, 16.99 versus 13.28 at 3B, and 24.96 versus 19.33 at 7B.
- Mixing skill and general data: With 30% SkillCorpus, agentic scores reach 38.60 with Dolmino and 37.58 with SmolLM, while general scores remain 58.64 and 58.83.Mixtures containing 20%–50% skill data outperform both endpoints under both general corpora; above 50%, agentic scores fall rapidly.
- Multi-file skill assembly: Reference Insert reaches a four-benchmark agentic score of 22.06, versus 20.19 for DeepSeek-Coder and 16.09 for Random File.Because Reference Insert and Random File contain the same files under the same budget, the 5.96 difference reflects file arrangement.
- Reinforcement learning: Under identical SFT and GRPO, SkillCorpus scores 11.94 versus 7.25 for Dolmino and 4.95 without mid-training.Its advantage remains after reinforcement learning, which provides no tool-use supervision.
Limitations and Ethical Considerations
The study’s scope is constrained by SkillCorpus’s single-repository provenance, limited evaluation coverage, and residual safety risks from executable workflows.
- Scope and coverage: SkillCorpus reflects the ClawHub community, with overrepresented programming languages, tool ecosystems, and task categories and underrepresented low-resource languages and specialized agentic domains.The experiments cover English and Chinese data, three 1.6B–7B backbones, and reported training budgets; larger models and other corpus compositions remain untested.
- Evaluation boundaries: The aggregate benchmarks do not separately measure tool selection, argument construction, error recovery, unsafe action sequences, or interactive long-horizon behavior.
- Safety considerations: Residual errors or unsafe instructions may remain despite removing detected personal data, secrets, binaries, encoded payloads, and benchmark-derived content.Because skills can specify executable workflows, downstream risk depends on the tools and permissions available to the model.
Conclusion
This work studies multi-file skill packages as mid-training data before behavior-oriented post-training and evaluates their effects across backbones and training configurations. SPT consistently improves agentic performance while largely preserving general performance, with gains remaining across mixtures, package organizations, and post-training methods.
- Conclusion: SPT uses skill packages during mid-training before behavior-oriented post-training to study downstream agentic capabilities.
- Conclusion: SPT consistently improves agentic performance over direct post-training and mid-training on general or trajectory data while largely preserving general performance.
- Conclusion: Mixture, assembly, and RL experiments show that the gains remain under different corpus compositions, package organizations, and post-training methods.
- Conclusion: Skill packages can provide reusable workflow knowledge during training, extending their role beyond inference-time context.
Corpus and Training Details
SkillCorpus is a cleaned, structured collection of multi-file reusable workflow packages, assembled and screened for training under controlled corpus, serialization, decontamination, and hardware settings. The experiments preserve package structure, describe corpus composition with proxies, and compare multiple backbones and post-training configurations.
- Corpus: SkillCorpus is sourced from ClawHub and contains trusted, audited multi-file reusable workflow packages spanning document, code, browser, spreadsheet, conversion, multimodal, and tool-adapter tasks.Category labels describe the corpus but are not used during training.
- Corpus accounting: The main mid-training comparison materializes Dolmino, AgentBank, and SkillCorpus to the same fixed token budget, with EOS padding included in exact post-packing totals.
- Composition proxies: AgentBank uses sampled-token shares and SkillCorpus uses package shares as corpus-native domain and tool-coverage proxies, while Unicode-script proportions proxy language composition.
- Serialization: The default serializer preserves package boundaries, file paths, and explicit separators, as illustrated by package, SKILL.md, metadata, and content sections.
- Filtering and cleaning: Cleaning combines manual audits and heuristic rules for semantic deduplication, generated content, erroneous workflows, structural validity, unsupported resources, encoded data, and possible secrets.Long sections are prioritized rather than randomly truncated, while packages shorter than 40 characters after cleaning are rejected.
- Decontamination: Benchmark decontamination removes entire packages containing benchmark-specific or derivative content after manual audit and model judgments, while exact 13-gram overlap is used as a review diagnostic.Approximately 0.3% of candidate packages are excluded from the training corpus.
Evaluation Protocols and Output Parsing
The evaluation combines four agentic benchmarks with six general benchmarks, using task-specific parsing, execution, exact-match, likelihood, and rubric-based metrics. Aggregate scores use fixed unweighted averages, while malformed or unparseable outputs are generally counted as incorrect.
- Agentic benchmarks: Agentic evaluation covers API invocation, tool selection, multi-step tasks, and tool-related reasoning across four benchmarks.The benchmarks are API-Bank, MetaTool, APTBench, and ToolEyes.
- General benchmarks: General capability is measured with six OLMES benchmarks using fixed five-shot likelihood-ranking evaluations.The benchmarks are ARC-Challenge, BoolQ, HellaSwag, PIQA, WinoGrande, and MMLU.
- Score aggregation: Aggregate benchmark scores are unweighted means, and unparseable outputs are counted as incorrect except for specified recovery procedures.APTBench averages eight SWE and eleven DR metrics before combining the domain scores directly.
- Output parsing: API-Bank, MetaTool, and APTBench score parsed outputs through execution, selection matching, or exact-match procedures, with invalid outputs scored incorrect.API-Bank uses execution-based evaluation; MetaTool uses Correct Selection Rate; APTBench applies task-specific parsers and metrics.
- Output parsing: ToolEyes evaluates five capability dimensions from ReAct interactions, combining format alignment, intent comprehension, behavior planning, tool selection, and answer organization.The overall score averages the five component scores and is multiplied by 100 for reporting.
Full Skill Mixture Ratio Results
The mixture-ratio sweep reports per-benchmark agentic results for SkillCorpus combined with either Dolmino or SmolLM, with best and second-best values marked within comparison blocks.
- Mixture-ratio results: Tables 12 and 13 provide full per-benchmark agentic and general results for SkillCorpus mixtures with Dolmino and SmolLM.Aggregate columns are omitted because the main figure reports agentic and general averages.
Full Skill-Aware Data Organization Results
The data-organization ablation compares serialization strategies under matched package, tokenizer, validation, separator, seed, and block-construction conditions. Reference-aware placement substantially reduces instruction-to-resource distance and produces the strongest reported agentic result without sacrificing general performance.
- Experimental setup: Five serialization strategies are compared from the same cleaned skill-package collection under matched preprocessing and 4,096-token block conditions.The comparison uses the same tokenizer, validation ratio, separators, and five-seed averaging protocol.
- Serialization strategies: Reference Insert places referenced supporting files near their first mentions, whereas alternative strategies preserve source order, shuffle files, add metadata, or apply fill-in-the-middle transformations.Reference Insert uses reference detection and reordering; the alternatives implement distinct generic packing schemes.
- Reference locality: 13,781.807 tokens, or 94.92%, is the average reduction in reference distance achieved by Reference Insert relative to original cleaned file order.Mean distance decreases from 14,518.996 to 737.189 MeCo-tokenizer tokens across 70,655 references in 17,957 packages.
Verifiable Rewards (RLVR)
The RLVR study tests SPT with a 360M backbone, Tulu 3 SFT, and GRPO-based reinforcement learning under shared evaluation settings. The appendix provides the corresponding hyperparameters and full agentic and general benchmark results, alongside planned release of the training artifacts.
- RLVR setup: RLVR experiments use SmolLM2-360M, Tulu 3 SFT, and GRPO inspired by DeepSeekMath through TRL.Tables 18 and 19 contain the full RLVR benchmark scores behind the main-paper summary.
- Reproducibility: The authors state that they will release corpus-construction, cleaning, decontamination, Reference Insert, configuration, and checkpoint artifacts.The planned release covers mid-training, post-training, and final checkpoints for the reported conditions.
- RLVR setup: Tables 16 and 17 document the mid-training, SFT, and GRPO hyperparameters for the SmolLM2-360M RLVR experiments.The evaluated configurations share identical RLVR settings.
- RLVR results: Tables 18 and 19 report full agentic and general benchmark results after RLVR post-training.The agentic-results table uses the SmolLM2-360M backbone.