Source-linked AI summary
SkillGen: Verified Inference-Time Agent Skill Synthesis
Yuchen Ma, Yue Huang, Han Bao, Haomin Zhuang, Swadheen Shukla, Michel Galley, Xiangliang Zhang, Stefan Feuerriegel
TL;DR
High-quality inference-time skills are still largely hand-written, and existing synthesis methods miss contrastive information from failures while lacking explicit net-effect verification. SkillGen synthesizes auditable skills from successful and failed trajectories through iterative generation, verification, and refinement, selecting candidates by paired performance comparisons. Across eight base LLMs and broad benchmarks, it improves held-out accuracy by +3.27 to +10.08 percentage points and produces skills that often transfer across models.
Problem
Existing skill synthesis relies heavily on hand-written skills and often misses contrastive success-failure signals or the regressions introduced by candidate skills.
Method
SkillGen uses multi-agent contrastive induction, iterative generation-verification-refinement, and paired verification to synthesize one auditable skill with positive net effect.
Results
Across eight base LLMs, SkillGen improves held-out average accuracy by +3.27 to +10.08 percentage points and consistently outperforms skill-generation baselines.
Takeaways & Limitations
SkillGen produces reusable inference-time skills that improve performance across diverse benchmarks and can transfer across models without parameter updates.
Takeaways & Limitations
For a fixed, non-adaptively chosen skill and i.i.d. verification instances, the empirical net-effect estimate is unbiased in expectation.
Abstract
from arXiv · showhide
Skills are a promising way to improve LLM agent capabilities without retraining, while keeping the added procedure reusable and controllable. However, high-quality skills are still largely written by hand. We introduce SkillGen, a multi-agent framework that synthesizes a single auditable skill from trajectories generated by a base agent. The output is a human-readable artifact that can be inspected before use. Rather than merely summarizing trajectories, SkillGen leverages contrastive induction over both successful and failed trajectories to identify reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures. SkillGen then generates candidate skills and iteratively refines the skill. A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance. Specifically, we compare outcomes on the same instances with and without the skill, so that we account for both repairs (cases where the skill fixes a baseline failure) and regressions (cases where the skill breaks a baseline success). Across a broad range of agents and datasets, SkillGen consistently improves held-out performance, outperforms existing skill-generation baselines, and produces skills that transfer across models.
1 Introduction
SkillGen addresses limitations in hand-written and success-focused skill synthesis by contrasting successful and failed trajectories and empirically verifying candidate skills' net effects. Its multi-agent framework produces auditable skills through iterative generation, verification, and refinement, improving held-out performance across models and benchmarks.
- High-quality inference-time skills remain largely hand-written despite being reusable, modular, and auditable procedures that avoid modifying model weights.Skills can be inspected, revised, and tested before deployment.
- Existing methods often learn from successful trajectories or summarize failures in isolation, missing contrasts between nearby successes and failures.Such contrasts can reveal correct intermediate behaviors that failed roll-outs omit.
- Skill synthesis must account for both repairs of baseline failures and regressions on baseline successes by comparing performance with and without each skill.This frames skill synthesis as an interventional problem requiring empirical net-effect verification.
- SkillGen uses contrastive induction and a generation-verification-refinement loop to derive a single readable skill from successful and failed trajectories.Specialized agents extract success patterns and failure modes, generate candidates, refine them with feedback, and select by held-out net effect.
- Across eight base LLMs, held-out average accuracy gains range from +3.27 to +10.08 percentage points, while SkillGen consistently outperforms skill-generation baselines.Ablations support contributions from contrastive induction, verification-guided refinement, and the verification gate; transfer analysis also evaluates cross-model generalization.
- SkillGen formulates automatic inference-time skill synthesis as producing one auditable skill that improves a base agent without parameter updates.The framework learns from both successful and failed trajectories, verifies positive net effect, and studies held-out gains and cross-model transfer.
2 Preliminaries
SkillGen treats skills as inference-time interventions that alter agent trajectories and outcomes, then estimates their net effect by comparing skill-augmented and baseline behavior on shared inputs. Its formal setup separates induction data from a verification subset and represents skills as structured, optionally executable and document-backed interventions.
- Inference-time skills are interventions whose effects are assessed by comparing outcomes with and without the skill on the same inputs.This comparison supports estimating how a skill changes task performance relative to baseline behavior.
- The data are split into an induction subset for analyzing agent behavior and a verification subset for evaluating and selecting candidate skills.The verification subset is used during construction-time candidate assessment.
- The base agent is modeled as a stochastic trajectory kernel, with the empty intervention defining no-skill behavior.A trajectory includes the sequence of LLM interactions, tool calls, observations, and final output.
- The task-level evaluator maps an instance and trajectory to a success probability, with observed outcomes sampled as Bernoulli variables.Deterministic evaluators are included as the special case where the evaluator returns only 0 or 1.
- A candidate skill consists of a structured prompt, task metadata, optional executable scripts, and optional auxiliary documents.Together these components define the skill space considered by SkillGen.
- Potential outcomes distinguish baseline performance from skill-augmented performance, allowing candidate interventions to be evaluated through their net effect.The framework accounts for both improvements and degradations on the same input distribution.
3 SKILLGEN
SKILLGEN compresses successful and failed trajectories into contrastive diagnostics, generates auditable candidate skills, and iteratively verifies and refines them against no-skill behavior on identical inputs.
- Framework overview: SKILLGEN uses baseline elicitation, contrastive induction, and an iterative generation–verification–refinement loop to produce a single auditable skill.The framework processes observed trajectories and returns a readable intervention whose empirical effect is evaluated before deployment.
- Contrastive behavioral induction: The diagnostic summary Z combines a task summary, clustered failure summaries, clustered success summaries, and local contrastive observations.These components convert variable-length trajectories into a lower-dimensional representation for skill generation.
- Contrastive behavioral induction: Local contrastive analysis compares nearby failed and successful rollouts to identify successful behaviors omitted from failures on the same task type.This anchors generated advice in procedures already demonstrated by the base agent.
- Candidate skill generation: Candidate skills encode task context, reusable success patterns, and failure-avoidance patterns derived from clustered and contrastive diagnostics.For tool-intensive tasks, candidates may also include scripts and reference documents, while later refinement edits only the natural-language body.
- Paired verification and refinement: Verification evaluates each candidate on the same instances with and without the skill, tracking repairs, regressions, and unresolved failures.The candidate with the largest construction-time net effect is selected, and structured feedback supports subsequent refinement.
4 Experiments
SKILLGEN improves held-out performance across models and benchmarks, with gains supported by paired evaluation, component ablations, baseline comparisons, transfer tests, and targeted analyses. Verification and refinement help retain beneficial interventions while limiting regressions.
- Evaluation protocol: Paired held-out evaluations compare no-skill and skill-augmented rollouts on the same task instances after construction is complete.Construction uses separate induction and verification subsets before deployment status is fixed.
- Main results: SKILLGEN improves average held-out accuracy for all eight base agents, with gains ranging from +3.27 to +10.08 percentage points.Across 80 benchmark–split–model entries, 50 improve, 25 remain unchanged, and 5 regress.
- Baseline comparison: SKILLGEN achieves the largest overall improvement across comparisons with four automatic skill-generation baselines.The comparison covers ALFWorld IOD, ALFWorld OOD, and ScienceWorld across three agent models.
- Ablations: The complete system achieves the best result on every dataset–model pair in the ablation study.The study indicates contributions from contrastive induction, refinement, verification, Failure Lessons, and task-specific skill structure.
- Cross-model transfer: Across 120 off-diagonal transfer comparisons, 70% are non-negative and 42% exceed +5 pp.Transferability depends on the skill-generating model; the strongest generator differs between ALFWorld and ScienceWorld.
- Benchmark-specific findings: Verification-gated skills improve τ-Bench retail by +5.3 pp on average and ChemLLMBench yield prediction by +16.1 pp across six models.Property prediction improves for only a small subset of agents, while rejected or unchanged candidates are not deployed.
- Refinement: At round 8, the latest candidate has expected ∆= −3.1 pp, whereas the best verified candidate reaches +8.1 pp.This supports best-of-K selection during refinement rather than deploying the latest candidate.
C.1 Model Details
Table 2 identifies the base-agent models used in the experiments and distinguishes open-weight models from proprietary models accessed through hosted APIs.
- Base-agent models are listed in Table 2, with open-weight models distinguished from proprietary hosted-API models.
C.2 Datasets and Splits
The evaluation uses disjoint skill-training and held-out test pools, with induction, verification, and benchmark-specific configurations controlling trajectory analysis and candidate deployment.
- Held-out test instances receive paired rollouts with and without the skill, using the same instance identifier and random seed.
- The skill-training data is divided into induction and construction-time verification subsets.
- Induction uses up to eight failure and success clusters, while contrastive analysis retains up to 20 nearest failure–success pairs.
- Candidate skills are verified on a 70/30 induction/verification split and accepted only when construction-time net gain meets the deployment threshold.
C.5 Token Cost Analysis
Skill construction is a one-time model–benchmark cost, while retrieval keeps skill-augmented inference in the same few-thousand-token range as ordinary calls.
- 5.6M tokens is the mean one-time construction budget, ranging from 2.2M on ScienceWorld to 10.2M on τ-Bench.Using GPT-5.4-Mini API prices, this corresponds to approximately $8.2 per generated skill.
- 5,919 tokens is the median skill-augmented call, while the largest absolute per-call average is 6,358 tokens on τ-Bench.
- The construction pipeline includes baseline trajectory collection, induction, generation, refinement, and verification as a one-time cost per model–benchmark pair.
- Experiments are executed through hosted LLM APIs, so provider-side accelerator and memory configurations are not exposed.
C.6 Skill-Generation Baselines
The baseline comparison adapts competing skill-generation methods to produce one fixed Markdown-formatted skill per benchmark–model pair and evaluates that intervention under a shared interface.
- Baseline methods: Trace2Skill analyzes success and error branches in parallel before consolidating their proposed patches through hierarchical LLM merging.
- Baseline methods: SkillX extracts skill cards from successful trajectories, clusters and merges them, filters low-quality cards, and canonicalizes the retained library into one skill.
- Baseline methods: EvoSkill maintains three candidate programs for four iterations and admits candidates that outperform the weakest frontier member on a fixed validation subset.
- Baseline methods: CoEvoSkills uses an information-isolated surrogate verifier that returns structured rollout diagnostics to the skill generator.
- Controlled comparison: All methods are evaluated as one fixed, auditable skill, avoiding extra test-time choices such as retrieval, routing, or stochastic skill selection.This controlled adaptation improves comparability but does not exhaust native multi-skill capabilities.
- Contrastive analysis: Failures cluster compactly while successes spread broadly in the ALFWorld t-SNE visualization, motivating contrastive failure–success pairing.
C.7 Evaluation Metrics and Gate-Off Handling
SkillGen evaluates each benchmark–split–model cell by comparing baseline and skill-augmented accuracy, while separately tracking repairs, regressions, and net gain. Deprecated skills reuse the no-skill baseline on the skill side.
- Each evaluated benchmark–split–model cell reports baseline accuracy, skill-augmented accuracy, and their paired difference in percentage points.The paired difference is defined as ∆ = accskill − accbase.
- Evaluation also records repairs, regressions, and net gain, defined as repairs minus regressions.Repairs are baseline-wrong cases fixed by the skill; regressions are baseline-correct cases broken by it.
- When construction-time verification deprecates a skill, evaluation reuses the no-skill baseline for the skill condition.
C.8 t-SNE Visualizations
Figure 8 visualizes SkillGen’s contrastive induction on ALFWorld using t-SNE for the gpt-5.4-nano model.
- Figure 8 presents a t-SNE visualization of SkillGen’s contrastive induction on ALFWorld.The visualization uses gpt-5.4-nano.
C.9 Failure Analysis
Skill augmentation can still fail through held-out regressions, incomplete procedures, incorrect chemistry grounding, missing algorithmic structure, and risks beyond the evaluated distribution. These analyses identify where skills help and where they remain bounded.
- Residual regression: Accepted skills can still regress on held-out instances because verification-subset benefits may overgeneralize.On ALFWorld OOD, Llama-3.1-8B fell from 67.45% to 65.10%; on ChemLLMBench yield prediction, Mistral-Nemo fell from 43.33% to 20.00%.
- Interactive environments: Interactive-environment failures often omit later prerequisites despite identifying a plausible initial subgoal.ALFWorld’s largest failure cluster contains 65 incomplete dependency-planning cases, while ScienceWorld includes 25 ungrounded-action, 20 incomplete-sequence, and 19 incomplete-goal-to-action cases.
- Chemistry tasks: Chemistry failures primarily involve grounding reaction roles and decision criteria rather than missing factual knowledge.For Qwen-2.5-7B on ChemLLMBench yield prediction, 27 of 30 training examples failed, including 14 superficial feasibility assessments and 13 reaction-role misparsings.
- Code generation: Skill augmentation cannot fully compensate for missing global problem structure in code generation.Qwen-2.5-7B had 113 failures out of 150 LiveCodeBench problems, dominated by incomplete algorithmic modeling, incomplete algorithm realization, and structure-mapping failure.
- Broader impacts: Verification checks are limited to the evaluated task distribution and should be paired with application-specific safety evaluation, human review, access controls, and ongoing monitoring.