PaperlayerView exact revision on arXiv

Source-linked AI summary

VCE-Skill: Enhancing Skill Self-Evolution with Version-Change Experience

Jianming Chen, Xuanbin Ye, Yawen Wang, Junjie Wang, Qing Wang, Fanjiang XU

arXiv:2608.16544v1cs.MAcs.AI

TL;DR

Public skill version histories have received limited attention as reusable experience, and their value beyond current-task trajectories remains unclear. VCE-Skill distills and fuses public changes with trajectory-derived proposals; across the evaluated benchmarks, models, and base evolvers, it improves skill self-evolution and cross-model transferability.

  • Problem

    Public skill version histories have received limited attention as reusable experience, and whether they add knowledge beyond current-task trajectories remains unclear.

    Source: §1 · p. 1

  • Method

    VCE-Skill distills public skill changes into reusable structured experience and adaptively fuses it with trajectory-derived proposals.

    Source: §1 · p. 2, §6 · p. 7

  • Results

    Across five benchmarks, four LLMs, and three base evolvers, VCE-Skill improves skill self-evolution and yields skills with stronger cross-model transferability in the evaluated setting.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    Public skill version histories are an underexplored source of prior knowledge for skill self-evolution.

    Source: §6 · p. 7

Abstract

from arXiv · show

Agents increasingly rely on reusable skills to encode task knowledge, tool-use procedures, and validation rules. Existing skill self-evolution methods primarily revise skills using execution trajectories collected from current tasks, leaving the evolution knowledge accumulated in public skill version histories largely untapped. Our pilot study reveals a clear complementarity between the two sources: public skill changes provide reusable evolution priors, whereas trajectories provide evidence grounded in the current task. Motivated by this, we propose VCE-Skill, which distills noisy and implementation-specific public skill changes into reusable, structured version-change experience and adaptively fuses it with trajectory-derived proposals from the base evolver, thereby exploiting external experience while retaining task-specific evidence. Extensive experiments demonstrate that VCE-Skill improves skill self-evolution, increasing mean scores by 3.20--4.98 points; transfer experiments further show that the resulting skills achieve stronger cross-model transfer performance. Our work highlights public skill version changes as a previously underexplored yet effective source of prior knowledge and advances trajectory-driven skill self-evolution.

1 Introduction

VCE-Skill treats public skill version histories as reusable evolution priors that complement current-task trajectories. It distills those histories into structured experience and adaptively fuses relevant experience with trajectory-derived proposals during skill optimization.

1 Introduction

  • Trajectory-driven evolution grounds revisions in task-specific execution evidence, but noisy or incomplete trajectories can produce unreliable guidance, overfitting, and missed reusable update knowledge (Mi et al. 2026; Huang et al. 2026).This leaves public skill version histories as a potential source of evolution knowledge beyond current executions.Source: §1 · p. 1
  • The motivation study finds complementary coverage: public histories capture broader changes, while trajectory-derived changes remain, so historical experience complements rather than replaces task-specific evidence.Raw diffs are repository-specific and may not fit current failures, motivating distillation and selective fusion.Source: §1 · p. 1
  • Experience Distillation turns raw diffs into structured update events, patterns, and insights; Adaptive Experience Attention selects relevant experience and fuses it with an independent trajectory-based self-proposal.Feedback from each optimization adjusts experience selection and attention allocation for the next iteration.Source: §1 · pp. 1–2
  • VCE-Skill consistently improves three base evolvers across five benchmarks and four LLMs, with experiments also examining ablations and cross-model transfer.The evaluation compares against two non-evolutionary baselines.Source: §1 · p. 2

2 Background and Related Work

The background defines agent skills and the execution-feedback loop used for skill self-evolution, then positions VCE-Skill as adding public version histories as a source of reusable evolution knowledge. It distinguishes skill-history mining from code-edit mining because skills are heterogeneous, multi-component artifacts.

2.1 Agent Skills

  • An agent skill is a discoverable, load-on-demand filesystem package whose directory contains at least a SKILL.md file.Its YAML metadata includes a required name and description; the SKILL.md instruction component is required, while scripts, references, configurations, templates, and other supporting components are optional. Agents inspect metadata for discovery, load instructions on activation, and access other components as needed.Source: §2.1 · p. 2

2.2 Skill Self-Evolution

  • Skill self-evolution typically executes tasks with a current skill, diagnoses trajectories, proposes an update, and evaluates it on held-out cases before accepting it (Shinn et al. 2023; Zhao et al. 2024).Existing methods derive update signals mainly from current or accumulated task interactions and evaluations, whereas VCE-Skill mines public skill version histories and integrates distilled experience with trajectory-derived proposals.Source: §2.2 · p. 2

2.3 Evolution Knowledge from Version Histories

  • Unlike code-history mining of syntax-level edits and repair contexts, VCE-Skill distills guidance from cross-repository skill changes and retrieves relevant experience to complement trajectory-derived proposals.Agent skills are heterogeneous, multi-component artifacts whose changes can encode semantic intents across instructions, scripts, references, and configurations.Source: §2.3 · p. 2

3 Motivation Study

Public skill version histories contain evolution knowledge beyond trajectory-derived changes, whose guidance can be limited by rollout quality and coverage. The study finds the sources complementary, motivating adaptive fusion of version-change experience with task-grounded trajectory evidence.

3 Motivation Study

  • Trajectory-based guidance can be limited by noisy or incomplete trajectories and by failure modes absent from sampled rollouts.Such evidence may obscure failure causes, offer limited actionable guidance, and leave useful update strategies unobserved.Source: §3 · p. 3

3.1 Data and Comparison Protocol

  • The comparison matched 400 public and 400 trajectory skill-change units within task domains, using shared component, intent, and pattern annotations with 100% of reviewed cases passing verification.Public changes came from adjacent GitHub and ClawHub versions; trajectory changes came from representative evolution methods.Source: §3.1 · p. 3

3.2 Key Observations

  • 372 of 400 BFCL units and 379 of 400 SearchQA units affect instructions; public changes add 5 BFCL intent categories and 4 pattern families, and 10 SearchQA intent categories and 6 patterns.Public changes are also distributed more broadly across components than trajectory-derived changes.Source: §3.2 · p. 3
  • The sources share core update categories, but public histories extend coverage beyond current rollouts, while BFCL trajectory-only intents and patterns show public histories cannot replace task-grounded evidence.This complementarity motivates integrating public version-change experience as an external prior through adaptive fusion.Source: §3.2 · p. 3

4 VCE-Skill Method

VCE-Skill has two stages: distilling public skill-version changes into a structured experience bank, then selecting relevant experience and adaptively combining it with trajectory-derived proposals to improve a target skill.

Change Abstraction.

  • VCE-Skill converts repository-specific version diffs into update events that separate each concrete diff from its affected component, semantic operation, and update intent.This structured representation supports abstraction into reusable version-change experience.Source: Change Abstraction · p. 4, §4.1 · p. 4

Experience Generation.

  • It groups related events into update patterns, synthesizes per-skill evolution insights, and generalizes cross-skill domain insights into transferable optimization principles.Entries from all collected skills are aggregated at these levels into an experience bank used during self-optimization.Source: Experience Generation · p. 4

Experience Selection.

  • For each iteration, VCE-Skill selects up to K experiences by comparing a context of the target task, current skill, and preceding feedback with experience-bank entries.Positive feedback favors external guidance; negative feedback makes selection more conservative, and updated skill state and feedback change later selections.Source: Experience Selection · p. 4

Attention Fusion for Skill Optimizer.

  • The base evolver proposes edits grounded in the current skill’s execution trajectory, while an LLM fuser combines them with external experience using source-level reliance weights.Both weights start at 0.5; they are prompt-level instructions, not probabilities, and the fuser prioritizes the higher-weight source when guidance conflicts.Source: Attention Fusion for Skill Optimizer · p. 5

Adaptation Feedback.

  • VCE-Skill updates source preference using attribution of realized proposal edits and validation-performance change, shifting attention toward the source associated with improving updates.External-supported realized edits receive +1 attribution, self-supported edits −1, and unrealized, unmatched, or mixed-provenance edits 0; attention updates use η = 0.1 and are clipped to [0.3, 0.7].Source: Adaptation Feedback · pp. 5–6

5 Experimental Evaluation

VCE-Skill improves every task score across base self-evolution methods and strengthens cross-model transfer. Ablations indicate that distilling version-change records and adaptively weighting external experience are important to these gains.

5.1 Experimental Setup

  • Evaluation spans five benchmarks and four agent models, comparing VCE-Skill with no-skill, one-shot, and iterative skill-evolution baselines.The benchmarks cover question answering, document and numerical reasoning, embodied execution, spreadsheet operations, and function calling; configurations were run three times and LLM-based judging was manually audited.Source: §5.1 · p. 6

5.3 Ablation Study.

  • Removing distillation drops performance below SkillOpt on all four models, while fixed-weight attention underperforms VCE-Skill but beats SkillOpt on three.Raw version-change records can contain nontransferable implementation details; adaptive attention adjusts the relative contributions of external experience and the self proposal as the skill evolves.Source: §5.3 · p. 6

5.2 Performance of VCE-Skill.

  • 3.20–4.98-point average-score gains, with improvements on every task for each base evolver, show VCE-Skill benefits multiple skill-evolution frameworks.The paired comparison reports mean results across three independent runs per configuration.Source: §5.2 · p. 6, §5.1 · p. 6, Table 1 · p. 7

5.4 Transferability of Skills.

  • 6.04, 5.25, and 5.15-point average-score gains on the three target models show stronger transfer for skills evolved with VCE-Skill than with SkillOpt.The reported transfer gains exceed gains over available paired non-VCE counterparts in Table 1; the authors attribute this to reusable experience that reduces source-model-specific behavior.Source: Table 2 · p. 7, §5.4 · p. 7

6 Conclusion … A.4 Broader Public-History Statistics

VCE-Skill combines reusable priors distilled from public skill histories with task-specific trajectory evidence, improving skill self-evolution and cross-model transfer in the evaluated setting. The supplementary analyses describe the comparison protocol, change taxonomy, examples, and broader public-history patterns.

6 Conclusion

  • Across five benchmarks, four LLMs, and three base evolvers, VCE-Skill improves skill self-evolution and produces skills with stronger cross-model transferability in the evaluated setting.It distills public changes into structured prior experience and adaptively fuses that experience with trajectory-derived proposals from the base evolver.Source: §6 · p. 7

Appendix for VCE-Skill

  • The appendix organizes supplementary material around the motivation study, VCE-Skill’s mechanism, and experimental details, including implementation, reproducibility, and extended analyses.Source: Appendix for VCE-Skill · p. 10

A.1 Matched Source Comparison

  • The matched comparison uses a shared skill-change unit and component, intent, and pattern taxonomy to compare public histories with trajectory-driven updates.The benchmark-matched corpus contains 400 units from each source; the public pools retain 38 skills and 1,266 BFCL updates, and 21 skills and 1,089 SearchQA updates, while the three trajectory methods contribute 1,200 updates total.Source: §A.1 · p. 10

A.2 Skill-Change Units and Annotation Taxonomy

  • Skill-change units capture edits serving one semantic intent through one reusable pattern; annotations label affected components, update intent, and modification pattern.Functionally coupled edits can span files or components, while distinct intents form separate units; formatting-only changes, generated files, and dependency-lock churn are excluded. Components are multi-label, while intent and pattern each receive one primary label.Source: §A.2 · p. 10

A.3 Representative Change Units

  • A representative public update coordinates guidance and code to maintain service compatibility, while a trajectory-derived update adds extraction rules for a SearchQA failure format.Source: §A.3 · p. 10

A.4 Broader Public-History Statistics

  • Instructions are the most frequently modified component in 11 of 12 surveyed domains, with within-domain shares of 58.9%–82.9%; scripts, references, and configurations also recur.The survey covers 36,719 substantive adjacent-version updates from 2,146 public skills across 12 domains.Source: §A.4 · p. 10

A.5 Additional Analyses of Trajectory-Driven Evolution … B.6 Base-Evolver Interfaces

VCE-Skill combines reusable strategies from public version histories with task-specific trajectory evidence through adaptive proposal fusion, while retaining each base evolver’s native workflow.

A.5 Additional Analyses of Trajectory-Driven Evolution

  • From low to high failure-mode coverage, the train–test gap fell from 2.28 to 1.04 and mean P2F count from 12.7 to 5.9; higher-utility trajectories also aligned with more positive validation changes.Negative updates clustered among low-utility trajectories, while high-utility trajectories more often supported positive changes.Source: §A.5 · pp. 10–11, Table 2 · p. 11, Figure 3 · p. 12

B.1 Offline Experience-Bank Construction

  • Adjacent-version diffs from 1,904 GitHub and 242 ClawHub skills are distilled through event, pattern, skill, and domain levels, retaining evidence-grounded, actionable, transferable guidance.Compatible or equivalent guidance is merged, while conflicting observations become conditional guidance only when applicability conditions explain the difference.Source: §B.1 · p. 13

Distillation Prompt Definitions

  • Diff extraction structures atomic events by evidence, affected components, operation, and intent; later prompts synthesize patterns into skill-level and cross-skill domain insights.Source: Distillation Prompt Definitions · p. 13

B.2 Online Optimization Algorithm

  • VCE-Skill changes proposal construction while leaving the base evolver’s candidate evaluation, retention, checkpointing, and stopping mechanisms native.Source: §B.2 · p. 13

B.3 Experience Selection and Aggregation

  • The selector chooses up to five relevant, actionable, nonredundant entries using current-skill gaps and prior feedback; if none qualify, the base self-proposal proceeds unchanged.Task and skill compatibility remain primary, and negative prior feedback demands stronger applicability evidence.Source: §B.3 · pp. 13–14

B.4 Attention Fusion and Skill Optimization

  • Fusion treats trajectory evidence as authoritative, using applicable external guidance to refine, constrain, or replace self edits when it identifies a concrete conflict or stronger strategy, and tracks provenance.Source: §B.4 · p. 14

B.5 Provenance Attribution and Adaptation Feedback

  • Attribution gives realized external edits +1, self edits -1, and mixed or unrealized edits 0; validation feedback shifts attention toward the source contributing more realized edits to improving updates.Unrealized edits include ambiguous, contradicted, wording-only, or unapplied changes.Source: §B.5 · p. 15

B.6 Base-Evolver Interfaces

  • A thin adapter preserves each base loop while translating proposals: EvoSkill and SkillClaw exchange complete skill documents, whereas SkillOpt fuses bounded patch edits.The native rollout, diagnosis, evaluation, retention, stopping, and checkpoint logic remains unchanged.Source: §B.6 · p. 15

B.7 End-to-End Evolution Case Study · C Additional Experimental Results and Analyses

The end-to-end case shows public version-change experience and trajectory-derived proposals combining into accepted Skill edits, alongside a measured hard-score gain. Supplementary experiments use a shared paired-comparison protocol across five benchmarks and four target models.

B.7 End-to-End Evolution Case Study

  • The public update distills recurring incomplete-output incidents into activation cues, completeness checks, and recovery rules reusable for structured-output skills.The case traces a public version change addressing incomplete structured outputs into reusable experience.Source: §B.7 · p. 16, Figure 12 · p. 19
  • The fuser combines three retrieved experiences with four self-proposed edits, and all four mixed-provenance edits appear in the accepted Skill update.The self-proposals address JSON closure, tool matching, output formatting, and pre-output validation.Source: §B.7 · p. 16
  • The representative parallel-call example improves from zero matched calls out of two to two matched calls out of two.Source: §B.7 · p. 16
  • The hard score rises from 0.5661 to 0.5740 across 507 paired validation examples, with 17 wrong-to-right and 13 right-to-wrong transitions.The Figure 12 case reports the score change; the accompanying paragraph gives the paired-example count and transition counts.Source: §B.7 · p. 16, Figure 12 · p. 19
  • Mixed provenance yields zero source feedback, so the next-round experience and self weights remain balanced at λexp = λself = 0.5.The case reports f S_t = +1 and f_t = 0 before the weights remain unchanged.Source: §B.7 · p. 16, Figure 12 · p. 19

C Additional Experimental Results and Analyses

  • Supplementary experiments follow the main-paper protocol across five benchmarks and four target models, with three independent runs and paired comparisons matched on instances, initial skill, budget, and target model.Compared methods include No Skill, LLM Skill, EvoSkill, SkillClaw, SkillOpt, and VCE-enhanced variants of three iterative self-evolvers.Source: C Additional Experimental Results and… · p. 20
  • GPT-5.5 handles motivation-study annotation and analysis, while VCE-Skill distillation and fusion use each configuration’s target model; experiments run on a TITAN RTX server with 32 GB RAM.All models are accessed through APIs; the server also has an Intel Core i7-10700 CPU.Source: C Additional Experimental Results and… · p. 20

C.1 Ablation Results Across Base Self-Evolvers

Across three base self-evolvers and four agent models, VCE-Skill’s experience distillation and adaptive attention provide complementary improvements. The complete method achieves the highest average score in all twelve base-evolver–model combinations.

C.1 Ablation Results Across Base Self-Evolvers

  • VCE-Skill achieves the highest average score in all twelve base-evolver–model combinations across five benchmarks.Figure 13 reports the average performance across the five benchmarks for each base self-evolver and agent model.Source: §C.1 · p. 20
  • Removing experience distillation causes the larger degradation in most settings, while removing adaptive attention consistently reduces performance.The ablations replace distilled experience with raw version-change records, or adaptive attention with fixed equal weights for external experience and self-generated proposals.Source: §C.1 · p. 20
  • Distillation and adaptive attention provide complementary gains that generalize across EvoSkill, SkillClaw, and SkillOpt.Distillation turns concrete version changes into reusable evolution knowledge, while adaptive attention controls external experience’s contribution during self-evolution; fixed equal weighting usually retains part of the improvement over the base evolver.Source: §C.1 · p. 20

C.2 Hyperparameter and Attention Sensitivity

The default configuration—K = 5, η = 0.1, and clipping bounds [0.3, 0.7]—achieves the highest score for each hyperparameter across all five benchmarks. Every evaluated configuration also outperforms the corresponding SkillOpt baseline on all five benchmarks.

C.2 Hyperparameter and Attention Sensitivity

  • η = 0.1 performs best, while η = 0.2 reduces performance by 2.28–3.88 points across five benchmarks.The experiments use SkillOpt as the base self-evolver and Qwen3.5-27B as the agent model.Source: §C.2 · p. 20
  • K = 4 performs within 0.02–0.11 points of K = 5, whereas K = 7 reduces performance by 1.11–1.88 points.The default selection budget is K = 5.Source: §C.2 · p. 20
  • Clipping bounds [0.4, 0.6] and [0.2, 0.8] stay within 0.29 points of the best result, while [0.1, 0.9] reduces performance by 0.71–1.37 points.The default clipping bounds are [0.3, 0.7].Source: §C.2 · p. 20
  • All evaluated configurations outperform SkillOpt on all five benchmarks; the results support K = 5, η = 0.1, and [0.3, 0.7] as defaults.The study attributes sufficient experience coverage to a moderate selection budget, and stable attention updates to moderately constrained clipping bounds.Source: §C.2 · p. 20

C.3 Complete Cross-Model Transfer Results

VCE-Skill improves cross-model transfer across every evaluated source model, target model, and benchmark. Because transferred skills receive no target-side evolution, the gains reflect stronger transferable guidance.

C.3 Complete Cross-Model Transfer Results

  • The evaluation tested 12 directed pairs among four models on five benchmarks, applying each evolved source skill to the other three target models without further evolution.The consistent gains across source models, target models, and benchmarks therefore reflect stronger transferable guidance rather than target-side evolution.Source: §C.3 · p. 21
  • 5.56 points overall mean gain, with improvements in all 60 benchmark-level comparisons and gains ranging from 5.15 to 6.19 points.SkillOpt+VCE-Skill outperformed SkillOpt for all 12 directed source–target pairs.Source: §C.3 · p. 21
  • Source-model average gains ranged from 5.37 to 5.71 points; GPT-5.2 received the largest target-model mean gain at 6.12 points, and every target improved by at least 5.23 points.Source gains were 5.48 for Qwen3.5-27B, 5.37 for GPT-5.2, 5.71 for DeepSeek-v3.2, and 5.69 for Claude Sonnet 5.Source: §C.3 · p. 21

D Human Audit of LLM-based Decisions

A human audit of six LLM-based semantic decisions found generally high acceptance and annotator agreement, supporting the reliability of the analyses and optimization procedure without implying error-free judgments.

Audit Scope and Sampling

  • The audit covered six decision types, sampling 200 instances per type with stratification and balancing tailored to the decision and experimental setting.Skill-Change Coding was balanced across domains and evolution sources; the remaining four types were stratified across tasks, target skills, iterations, and base evolvers, while Experience Generation samples covered update patterns, evolution insights, and domain insights.Source: D Human Audit of LLM-based Decisions · p. 22

Annotation Process and Metrics

  • Two annotators independently judged anonymized instances, disagreements went to a third annotator, and pilot guidelines were repeated if κ fell below 0.7.The audit reports adjudicated Human Acceptance Rate and pre-adjudication Cohen’s κ between the initial annotators.Source: D Human Audit of LLM-based Decisions · pp. 22–23

Annotation Criteria

  • Taxonomy, coding, and abstraction decisions were accepted only when their labels and semantic claims were coherent, correctly supported, and faithful to the underlying changes.Taxonomy criteria also required distinguishable, source-independent categories; coding required grouping coupled edits with the same intent and pattern while excluding unrelated edits; abstraction required the event to match the diff.Source: D Human Audit of LLM-based Decisions · p. 22
  • Experience decisions were assessed for faithful, reusable guidance, applicability to the current task and skill state, and correct provenance attribution.Generation entries had to summarize supporting items without unsupported claims; attribution had to match the realized change and label external, self-supported, or uncertain, unrealized, or mixed provenance as defined.Source: D Human Audit of LLM-based Decisions · pp. 22–23

Audit Results and Summary

  • HAR ranged from 86.5% to 100.0% and Cohen’s κ from 0.75 to 0.84 across six audited decisions, with outputs generally supported by human assessment but not error-free.Table 6 reports the audit results; the authors interpret them as evidence that the decisions are sufficiently reliable for the analyses and optimization procedure.Source: Table 6 · p. 23, D Human Audit of LLM-based Decisions · p. 23
  • Motivation-study Taxonomy achieved HAR 94.0% and κ 0.84, while all audited Skill-Change Coding instances were accepted after adjudication, with κ 0.84.These findings support the taxonomy construction and skill-change annotations used in the motivation-study findings.Source: D Human Audit of LLM-based Decisions · p. 23
  • Experience Selection had the lowest HAR, 86.5%, while Source Attribution achieved HAR 90.5% and κ 0.80.Selection’s κ was 0.75; the authors relate its lower acceptance to the context-dependent judgment of applicability and cite attribution results as support for its use in adaptive experience attention.Source: D Human Audit of LLM-based Decisions · p. 23

Related papers

Loading 2608.16544v1…