PaperlayerView exact revision on arXiv

Source-linked AI summary

From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills

Zisu Huang, Jingwen Xu, Yifan Yang, Ziyang Gong, Qihao Yang, Muzhao Tian, Xiaohua Wang, Changze Lv, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Xue Yang, Dongdong Chen, Xiaoqing Zheng, Chong Luo

arXiv:2605.23899v1cs.AI

TL;DR

Existing evaluations do not comprehensively explain whether model-generated skills work across their full lifecycle. This paper studies that lifecycle across domains, extractors, and targets, finding average benefit alongside substantial variance and 25% negative transfer, while a meta-skill improves extraction in all evaluated cells.

  • Problem

    Evaluation and understanding of model-generated skills remain limited across experience generation, skill extraction, and skill consumption, despite growing methodological momentum.

  • Method

    The paper conducts a utility-grounded study across five domains, using a minimal unified extraction framework and validating a rubric-derived meta-skill for extraction.

  • Results

    25% of evaluated extractor–target pairings exhibit negative transfer, while skill utility varies sharply across extractors and targets rather than following model scale or baseline strength.

  • Takeaways & Limitations

    The validated meta-skill prior improves extraction in all evaluated cells and plugs directly into any extractor’s system prompt.

  • Takeaways & Limitations

    The study consolidates each target’s experience into one domain-level skill supplied through the system prompt, leaving richer harnesses and larger skill libraries for future work.

Abstract

from arXiv · show

Language agents increasingly improve by reusing \emph{skills} -- structured procedural artifacts distilled from past experience. In particular, \emph{domain-level} and \emph{model-generated} skills are especially promising. They offer fast adaptation within a domain by encoding domain-specific recurring procedures, and they scale beyond labor-intensive hand-crafting. However, while extraction methods continue to proliferate, understanding remains limited, with no comprehensive study spanning the full skill lifecycle -- \textbf{experience generation}, \textbf{skill extraction}, and \textbf{skill consumption} -- to ask whether such skills actually work, when they work, and what makes them succeed or fail. To close this gap, we build a utility-grounded evaluation framework that provides systematic experimental results across extractors and target agents, covering five diverse agentic task domains. We find that model-generated skills are beneficial on average but exhibit non-trivial negative transfer, and that neither extractors nor targets behave uniformly. A model can be a strong extractor yet a weak consumer, or vice versa, with skill utility independent of model scale or baseline task strength. To explain these patterns, we then dissect each lifecycle stage in depth, analyzing how experience composition shapes skill quality, what properties characterize useful skills, and how the same skill transfers across different consumers. Finally, we translate these findings into a concrete \emph{meta-skill} that guides skill extraction toward the features tied to actual utility, which consistently improves skill quality across domains and substantially reduces negative transfer.

1 Introduction

This study examines model-generated, domain-level skills across the full lifecycle of experience generation, skill extraction, and skill consumption. It seeks to replace heuristic practice with utility-grounded principles for obtaining skills that are stable and effective in real agent systems.

  • Motivation: Skills are short, structured procedural artifacts distilled from past trajectories and loaded at inference time without retraining.Domain-level skills package recurring domain procedures into reusable artifacts, enabling adaptation to new tasks within a domain.
  • Motivation: Model-generated skill methods produce skills automatically at scale, either by distilling execution logs or iteratively refining multi-file packages with a verifier.These approaches offer scalability and automated iteration, while sharing a lifecycle that begins with execution-trajectory generation.
  • Study scope: The study evaluates the full trajectory-to-skill lifecycle through experience generation, skill extraction, and skill consumption.Its pipeline generates an experience pool, distills it into a single domain-level skill with minimal framework design, and evaluates downstream use.
  • Contribution: The broader aim is to move skill development from heuristic, intuition-driven practice toward a principled, utility-grounded discipline.The study is intended to help practitioners obtain skills that are stable and effective when deployed in real agent systems.

2 Related Work

Prior work frames agent skills as reusable packages extracted from experience and develops benchmarks for testing their downstream value. However, the field lacks systematic coverage of the full trajectory-to-skill lifecycle, which this work evaluates across extractors, consumers, and domains.

  • Automatic Generation of Reusable Knowledge from Agent Experience: Agent skills are composable packages of instructions, code, and resources loaded on demand to extend LLM capabilities without retraining.Recent surveys motivate automatic extraction of reusable skills from execution trajectories.
  • Benchmarks for Agent Skills: Existing benchmarks primarily test whether curated or discovered skills improve downstream performance over a no-skill baseline.SkillsBench, SWE-Skills-Bench, and Liu et al. probe this question, while SkillCraft studies composition and accumulation through test-time extraction and reuse.
  • Research Gap and Contribution: The field lacks systematic understanding of raw experience generation, skill extraction, and skill consumption across the full trajectory-to-skill lifecycle.This work addresses the gap with a comprehensive framework spanning skill extractors, skill consumers, domains, and detailed lifecycle-stage analysis.

3 Evaluation Framework

The evaluation framework models skill improvement as a three-stage lifecycle: experience generation, skill extraction, and skill consumption on held-out tasks. It measures utility through downstream performance gains, separating extractor efficacy from target evolvability while retaining extractor–target interactions.

  • Lifecycle: The lifecycle has three stages: M generates successful and failed trajectories, E extracts structured skills, and M consumes them on held-out tasks.This deployment-realistic protocol feeds skills distilled from M’s own interaction logs back to the same model, while varying E enables controlled extractor comparisons.
  • Extraction framework: The unified extractor uses minimal structure, leaving abstraction decisions to the model and retaining only per-trajectory extraction followed by hierarchical consolidation.Trajectories are processed independently into success and failure patterns, then grouped and merged by deduplication, generalization, and reconciliation.
  • Utility evaluation: Extracted skills are evaluated by downstream performance gain rather than text quality, using the same held-out split for baseline and skill-augmented evaluations.Positive delta indicates improvement, whereas negative delta indicates negative transfer.
  • Utility evaluation: Extraction efficacy measures how reliably a fixed extractor improves performance across target-specific experience pools.It captures the extractor-side effect of converting different targets’ experiences into useful skills.
  • Utility evaluation: Target evolvability measures how much a fixed target improves when different extractors distill skills from and feed them back to that target.The framework reports both metrics per domain and retains each extractor–target delta for interaction analysis.

4 Main Experiments

The main experiments evaluate model-generated skills across five domains, six target models, and five extractors, finding that skills improve performance on average but exhibit substantial, target- and domain-dependent negative transfer.

  • Experimental setup: The evaluation spans five qualitatively different domains, six target models, and five extractor models to characterize skill utility across extractor–target–domain combinations.The domains are embodied interaction, productivity software, software engineering, web search, and tool calling.
  • Overall results: 75% of evaluated entries improve downstream performance, but 25% have ∆ < 0, showing that model-generated skills can cause negative transfer.Negative transfer is common despite positive average gains.
  • Overall results: 47% of ALFWorld entries exhibit negative transfer, compared with 13% for SpreadsheetBench and SWE-bench-Verified, demonstrating strong domain dependence.ALFWorld is the most fragile domain, while SpreadsheetBench and SWE-bench-Verified have the lowest negative rates.
  • Extractor–target asymmetry: On SpreadsheetBench, Gemini-3.1-Flash-Lite achieves the highest EE while GPT-5.4 ranks last despite the strongest baseline, separating extraction from execution.Extractor-side performance does not simply follow model scale or baseline task strength.
  • Extractor–target asymmetry: On ALFWorld, GPT-5.4 benefits from all five extractors with TE = +4.93, whereas Gemini-3.1-Flash-Lite, Qwen3.5-35B, and Qwen3.5-9B have negative TE.The same extractors can produce different gains across targets, indicating that target experience shapes skill utility.

5 Diving Deeper into the Agent Skill Lifecycle

Downstream skill utility is shaped by all three lifecycle stages: experience composition, skill content beyond surface form, and target-specific consumption. Skills can transfer unevenly across consumers, with gains varying sharply and sometimes becoming negative.

  • Experience generation: Experience-pool success–failure composition strongly influences skill quality, while the optimal mix is domain-dependent and pure-failure pools are consistently worst.Successful trajectories provide workable procedures, whereas failures expose constraints and pitfalls.
  • Skill extraction: Skill format has no significant effect on any target (all p>0.34), whereas changing the extractor has a discernible effect on 5/6 targets (p<0.01).This contrast indicates that utility variance is driven by skill content rather than presentation format.
  • Skill extraction: 46.4%: LLM selection accuracy from skill text alone is indistinguishable from random, and falls to 15.8% for pairs with δ≥5%.Textual plausibility therefore often diverges from actual downstream utility.
  • Skill consumption: With identical skill text, gains range from +1.8 on Gem-3.1-Pro to +9.5 on Qwen3.5-35B, showing that consumption ability is target-specific.Some targets benefit strongly, while others see little effect or regress as skills reshape their default policies.

6 From Diagnosis to Intervention: Meta-Skill Guided Extraction

The section develops a utility-grounded, automated rubric-discovery pipeline and validates rubric dimensions that distinguish higher- from lower-utility skills. Operationalized as a meta-skill in extractor prompts, the validated rubric improves generated-skill quality across all evaluated cells, whereas a generic plausibility rubric harms performance.

  • Rubric Discovery and Validation: The automated pipeline derives candidate rubric dimensions from high-gap skill pairs, comparing per-pair differences between higher- and lower-utility skills before iterative consolidation.The resulting seven dimensions constitute the raw rubric, which is then validated for downstream utility.
  • Meta-Skill Guided Extraction: −0.59 pp was the plausibility rubric’s average performance change, reducing accuracy in 6 of 9 cells, whereas the validated rubric improved all nine cells (+1.55 pp average).The validated rubric was inserted as a compact generation-time meta-skill in the extractor’s system prompt and compared with unguided and plausibility-rubric prompts.
  • Meta-Skill Guided Extraction: The closed-loop intervention translates diagnostic findings into validated rubric dimensions and measurable downstream improvements in skill extraction.This demonstrates that utility-grounded benchmarking can guide both skill evaluation and extraction-system design through a drop-in meta-skill.

7 Conclusion … B.5 Extraction Hyperparameters

The paper presents a utility-grounded study of model-generated skills across experience generation, extraction, and consumption, finding average benefits alongside substantial variance and negative transfer. It also documents the experimental procedures, prompts, injection protocols, and default extraction settings used to study these effects.

  • 7 Conclusion: Model-generated skills help on average but show substantial variance and non-trivial negative transfer, with neither model scale nor textual plausibility reliably predicting utility.The study spans five diverse domains and multiple extractors and targets, and analyzes all three lifecycle stages to explain this variance.
  • A Limitations, Future Work, and Broader Impact: The evaluation prioritizes interpretability by consolidating each target’s experience into one domain-level skill supplied directly through the system prompt.This setup isolates performance changes attributable to the skill rather than retrieval policies, agentic scaffolding, or other pipeline components.
  • A Limitations, Future Work, and Broader Impact: Utility-grounded evaluation, a validated rubric, and a meta-skill prior help practitioners identify fluent skills that transfer poorly and avoid shipping skills that degrade performance.The same screening process can also save compute otherwise spent on unhelpful skills.
  • B.1 Experience Pool Collection: For each target-domain pair, the target model runs on the training split for multiple rounds, collecting successful and failed trajectories with domain-dependent pool sizes.Pool sizes reflect differences in training-split size and per-task cost.
  • B.2 Evaluation Details: Each domain is split 1:1 into experience-generation and held-out test instances, with shared splits across extractor-target combinations to attribute Δ differences to extraction rather than evaluation noise.Official benchmark training splits are sampled proportionally; otherwise available instances are randomly partitioned with a fixed seed.
  • B.3 Extraction Prompt Templates: The extraction framework analyzes trajectories into success and failure patterns, hierarchically consolidates pattern sets, and synthesizes schema-conformant skills through tool calls.Optional meta-skill guidance appends either a validated 3-dimension rubric or a 7-dimension plausibility rubric to extraction prompts.
  • B.4 Injection Template: Single skills are inlined in the target system prompt, whereas multi-skill libraries use progressive disclosure through list_skills, view_skill, and read_skill_file.SpreadsheetBench uses fenced skill blocks instead of OpenAI function calls.
  • B.5 Extraction Hyperparameters: The default mode-based extractor analyzes up to 3 behavioral modes per trajectory, groups modes in batches of 10, and iteratively merges them into one consolidated skill set.Extractor, experience pool, and target model vary across experimental conditions.

B.6 Compute Resource

The study accesses closed-source models through provider APIs and serves open-source models locally. A single node with 8 NVIDIA B200 GPUs supports all reported open-source extractor and target inference scales.

  • Model serving: Closed-source models are accessed through their respective providers’ APIs.The models are GPT-5.4, GPT-5.4-mini, Gemini-3.1-Pro, and Gemini-3.1-FL.
  • Model serving: Open-source models are served locally with vLLM on a single node equipped with 8 NVIDIA B200 GPUs.This setup is sufficient for all open-source extractors and targets used in the study at the reported inference scales.

C Format Normalization Experiment

Rewriting the strongest SpreadsheetBench extractor’s skill into four canonical output formats did not measurably change utility across targets. By contrast, extractor choice produced significant effects for most targets, indicating that content source mattered more than presentation format.

  • Experimental setup: The skill was rewritten as an ordered list, unordered list, checklist, or prose while preserving semantic content.GPT-5.4 performed each rewrite, and a verification pass checked content preservation and format compliance.
  • Experimental setup: Each format was evaluated on the same Section 4 test set for 3 independent rounds.
  • Statistical analysis: The Friedman test treated task instances as blocks and formats or extractors as treatments, testing whether all treatments produced equal performance.Effect size was measured with σ-ratio = σfactor/σround; values above 1 indicate factor effects exceeding run-to-run sampling noise.
  • Results: Format had no detectable effect on any target, with all p > 0.34 and all σ-ratios below 1.The extractor control instead produced significant effects for 5/6 targets, with p < 0.005 and σ-ratios well above 1.

D Behavioral impact analysis · E Pairwise Skill Evaluation

Skill consumption changes models’ task framing, exploration, and tool use, helping GPT-5.4 while producing mixed effects for Qwen3.5-9B. Pairwise evaluation compares substantially separated extractor outcomes using randomized, majority-voted judgments without an explicit quality rubric.

  • D Behavioral impact analysis: Skill consumption changes initial task framing, with GPT-5.4 shifting from formula answers toward Python computation and writing final values.This is especially helpful for cell-level tasks, where formula-based answers may appear reasonable but be unstable under evaluation.
  • D Behavioral impact analysis: For Qwen3.5-9B, skills encourage workbook-native workflows that can improve structural correctness but increase execution mistakes on fine-grained tasks.The added complexity can produce mixed outcomes: better structure on sheet-level tasks, but more failures on fine-grained tasks.
  • D Behavioral impact analysis: Both models more often inspect sheet structure, headers, used ranges, anchors, and target areas before editing.Skill consumption therefore changes how models build workbook understanding, not only their final actions.
  • D Behavioral impact analysis: GPT-5.4’s exploration changes slightly but usefully, whereas Qwen3.5-9B explores more yet can produce longer reasoning without correct execution.GPT-5.4 relies less on layout guesses, while Qwen3.5-9B’s extra exploration does not consistently improve results.
  • D Behavioral impact analysis: Consumed skills are usually absorbed into the prompt, changing use of existing tools rather than prompting explicit skill calls.GPT-5.4 more often uses bounded write-back, anchor-based addressing, and simple post-write checks.
  • D Behavioral impact analysis: Qwen3.5-9B shifts from pandas-style round-trip rewriting to openpyxl-based in-place editing, preserving structure while increasing workflow failure opportunities.The change is larger than GPT-5.4’s and reflects a tradeoff between workbook preservation and reliable execution.
  • E Pairwise Skill Evaluation: The unguided pairwise evaluation uses GPT-5.4, 9 independent votes per pair, majority voting, and randomized skill presentation order.Randomization is used to mitigate position bias.
  • E Pairwise Skill Evaluation: The judge receives only a one-sentence domain description and no evaluation rubric or quality criteria, relying on its implicit notion of skill quality.This design withholds explicit standards from the judge.

F Alternative Harness Evaluation · G Meta-Skill Guidance

Alternative interactive harnesses corroborate the main finding that skills provide modest average gains but highly variable transfer across targets. Meta-skill guidance improves downstream utility when based on screened, utility-validated dimensions, whereas an unscreened plausibility rubric hurts on average.

  • F Alternative Harness Evaluation: Claude Code and Codex re-evaluate SpreadsheetBench conditions through interactive tool use, independently testing whether Python-script harnesses explain the main results.These harnesses execute spreadsheet tasks interactively rather than through a fixed script.
  • F Alternative Harness Evaluation: ∆= + 0.4 pp average gain from skill injection persists across alternative harnesses, despite substantial variance across targets.The result is described as consistent with the main experiments.
  • F Alternative Harness Evaluation: Stronger targets, CC Opus and Codex GPT-5.4, show positive transfer from GPT-5.4-extracted skills.This pattern supports a target-side consumption-ability gradient.
  • F Alternative Harness Evaluation: The weakest target, Codex GPT-5.4-mini, shows no benefit from GPT-5.4-extracted skills.This echoes the consumption-ability gradient observed in the main experiments.
  • G Meta-Skill Guidance: Table 11 provides per-cell accuracy values for the no-skill baseline and original un-guided skill condition underlying Figure 5.These conditions establish the comparison for evaluating meta-skill guidance.
  • G Meta-Skill Guidance: The plausibility-based rubric using all 7 dimensions without screening hurts downstream skill utility on average.The table contrasts this rubric with a utility-validated alternative.
  • G Meta-Skill Guidance: The utility-validated rubric using 3 screened dimensions improves downstream skill utility in all nine cells.The table reports accuracy percentages for this guidance condition.

H Contrastive Skill Analysis … H.3 Representative Contrastive Cases

The section contrasts unguided plausibility judgments with utility-grounded skill analysis, then illustrates how actionable, domain-specific content distinguishes stronger skills from generic advice. It culminates in validated rubric dimensions and representative best-versus-worst cases across two domains.

  • H.1 Plausibility Rubric (Naive Baseline): GPT-5.4 generated seven plausibility criteria without skill-pair or downstream-utility data, capturing surface textual qualities rather than utility-grounded properties.These dimensions form the unguided baseline used in Section 6.
  • H.2 Raw Rubric (from Contrastive Pipeline): The contrastive pipeline synthesized seven candidate quality dimensions from recurring themes across 17 high-gap skill pairs.Better-rate measures the proportion of pairs where the higher-∆skill receives more favorable judgments on a dimension.
  • H.3 Representative Contrastive Cases: Representative best-versus-worst skill pairs are presented for SpreadsheetBench and ALFWorld with full skill text and highlighted key passages.The cases compare skills within two distinct agentic task domains.
  • H.3 Representative Contrastive Cases: Green highlights identify domain-specific failure mechanisms or executable countermeasures in the representative skill pairs.These markings emphasize passages with concrete leverage for task execution.
  • H.3 Representative Contrastive Cases: Red highlights identify generic advice that provides little actionable leverage in the representative contrastive cases.The visual contrast distinguishes concrete guidance from broadly stated recommendations.
  • H.2 Raw Rubric (from Contrastive Pipeline): The raw rubric’s seven dimensions include definitions and per-dimension better-rates measuring alignment with downstream utility.Three bold dimensions form the validated rubric used for guided evaluation and meta-skill extraction.
  • H.3 Representative Contrastive Cases: The SpreadsheetBench case uses GPT-5.4 as target and reports a ∆gap of 10.3 pp.This is one of the representative best-versus-worst comparisons shown in the contrastive analysis.
  • H.3 Representative Contrastive Cases: The ALFWorld case uses GPT-5.4 as target and reports a ∆gap of 6.0 pp.This provides a second representative contrastive comparison in a different task domain.
Loading 2605.23899v1…