PaperlayerView exact revision on arXiv

Source-linked AI summary

VCE-Skill: Enhancing Skill Self-Evolution with Version-Change Experience

Jianming Chen, Xuanbin Ye, Yawen Wang, Junjie Wang, Qing Wang, Fanjiang XU

arXiv:2608.16544v1cs.MAcs.AI

TL;DR

Current trajectory-driven skill evolution can miss broadly useful update knowledge, while public skill version histories remain underused and their added value is unclear. VCE-Skill distills public changes into structured experience and adaptively combines it with trajectory-derived proposals, improving self-evolution and cross-model transfer across the evaluated setting.

  • Problem

    Trajectory-driven evolution is bounded by current-task evidence, while public skill histories remain underused and their knowledge beyond trajectories is unclear.

  • Method

    VCE-Skill distills public version changes into reusable structured experience and adaptively fuses it with trajectory-derived proposals during skill optimization.

  • Results

    VCE-Skill improves skill self-evolution across five benchmarks, four LLMs, and three base evolvers, while producing skills with stronger cross-model transferability.

  • Takeaways & Limitations

    Public skill version histories provide complementary reusable prior knowledge that can augment, but not replace, task-specific trajectory evidence for skill evolution.

Abstract

from arXiv · show

Agents increasingly rely on reusable skills to encode task knowledge, tool-use procedures, and validation rules. Existing skill self-evolution methods primarily revise skills using execution trajectories collected from current tasks, leaving the evolution knowledge accumulated in public skill version histories largely untapped. Our pilot study reveals a clear complementarity between the two sources: public skill changes provide reusable evolution priors, whereas trajectories provide evidence grounded in the current task. Motivated by this, we propose VCE-Skill, which distills noisy and implementation-specific public skill changes into reusable, structured version-change experience and adaptively fuses it with trajectory-derived proposals from the base evolver, thereby exploiting external experience while retaining task-specific evidence. Extensive experiments demonstrate that VCE-Skill improves skill self-evolution, increasing mean scores by 3.20--4.98 points; transfer experiments further show that the resulting skills achieve stronger cross-model transfer performance. Our work highlights public skill version changes as a previously underexplored yet effective source of prior knowledge and advances trajectory-driven skill self-evolution.

1 Introduction

VCE-Skill addresses the limits of trajectory-driven skill evolution by extracting reusable version-change experience from public histories and adaptively fusing it with trajectory-derived proposals. Across 5 benchmarks and 4 LLMs, it consistently improves base evolvers and supports cross-model transfer evaluation.

  • Motivation: Trajectory-driven evolution grounds updates in task-specific execution evidence but can produce unreliable or overfitted guidance when trajectories are noisy, low-quality, or incomplete.Its guidance is bounded by the quality and coverage of trajectories collected in the current run.
  • Motivation: Public skill version histories provide a complementary source of reusable update knowledge by preserving revisions to instructions, scripts, and other components.These histories may encode update strategies absent from current task executions, although they have received limited attention.
  • Motivation: The motivation study finds complementary coverage: trajectory-derived changes concentrate on instructions and narrow intents, while public histories cover broader changes and trajectory-only changes remain.Thus, public histories complement rather than replace trajectory-derived evidence.
  • Method: VCE-Skill distills raw diffs into structured update events, patterns, and insights, then adaptively selects and fuses relevant experience with a base evolver’s trajectory-derived self-proposal.Adaptation feedback updates experience selection and attention allocation after each optimization iteration.
  • Evaluation: 5 benchmarks and 4 LLMs evaluate VCE-Skill by enhancing 3 skill self-evolution methods against 2 non-evolutionary baselines, including effectiveness, ablation, and cross-model transferability.The results show consistent improvement over the base evolvers.

2 Background and Related Work

This section defines agent skills as reusable filesystem-based capability packages and frames skill self-evolution as an execution-feedback loop. It reviews trajectory-driven methods and motivates studying version histories for heterogeneous, semantic skill changes.

  • Agent skills: An agent skill is a reusable filesystem-based capability package discovered and loaded on demand, with at minimum a directory containing SKILL.md.Skills may include procedural instructions, scripts, references, configurations, templates, and other supporting artifacts.
  • Skill self-evolution: Skill self-evolution executes tasks with the current skill, records trajectories, diagnoses behavior, proposes an update, and validates it before acceptance.If the candidate fails validation criteria, the current skill is retained.
  • Related methods: Recent methods implement trajectory-driven evolution through failure diagnosis, scored rollouts, task-level evaluation, verifiers, multi-file generation, or aggregation across users.Examples include EvoSkill, SkillForge, SkillOpt, Skills-Coach, CoEvoSkills, and SkillClaw.
  • Version-history mining: Prior version-history mining recovers recurring code-change and repair patterns, but agent skills require modeling semantic changes across heterogeneous components rather than syntax-level edits.Skill components include instructions, scripts, references, and configurations.

3 Motivation Study

The motivation study finds that public skill version histories provide evolution knowledge beyond trajectory-derived updates, while trajectory evidence remains grounded in current task rollouts. This complementarity motivates adaptively fusing public version-change experience with trajectory-based evolution.

  • Limitations of Trajectory Evidence: Trajectory-based guidance is limited by noisy or incomplete trajectories and by failure modes exposed only in sampled rollouts.These limitations can obscure failure causes and leave useful update strategies unobserved.
  • Comparison Design: The study compares public GitHub and ClawHub version changes with changes produced by EvoSkill, SkillClaw, and SkillOpt under a unified protocol.The comparison corpus contains 400 skill-change units from each source, with edits grouped by modification intent and pattern.
  • Complementary Coverage: 372 of 400 BFCL units and 379 of 400 SearchQA units in the trajectory source affect instructions, whereas public changes span components more broadly.The unified taxonomy annotates each change by component, intent, and reusable modification pattern.
  • Complementary Coverage: Public changes contain 5 public-only intent categories and 4 public-only pattern families for BFCL, plus 10 public-only intent categories for SearchQA.These findings show that public histories add semantic evolution knowledge absent from trajectory-derived changes.
  • Design Implication: Public histories and trajectory evolution share core update categories but each also contains unique intents and patterns, so neither source can replace the other.BFCL includes trajectory-only intents and patterns, while public changes extend evolution beyond current rollouts.

4 VCE-Skill Method

VCE-Skill distills public skill version histories into reusable, structured experience and adaptively combines task-relevant external guidance with trajectory-derived self-proposals. Iterative feedback from realized edits and validation performance adjusts future experience selection and source preference.

  • Framework overview: VCE-Skill has two stages: Experience Distillation builds a structured experience bank, while Adaptive Experience Attention selects and combines relevant experience with the current evolution proposal.The framework transfers experience from historical skill versions to guide target-skill self-evolution.
  • Experience distillation: Version diffs become update events that separate raw changes from affected components, semantic operations, and update intents, enabling repository-independent evolution guidance.Adjacent versions are compared chronologically, and each diff is decomposed into individual structured events.
  • Experience distillation: VCE-Skill abstracts events into update patterns, skill-level evolution insights, and cross-skill domain insights, then aggregates entries into an experience bank for task-relevant retrieval.The bank spans multiple skills and task domains and serves as an external knowledge corpus during optimization.
  • Adaptive experience attention: At each iteration, an LLM selector retrieves at most K experiences using the target task, current skill, and prior feedback, while the base evolver proposes edits from the execution trajectory.Positive feedback favors external guidance, whereas negative feedback makes external selection more conservative.
  • Adaptive experience attention: The fuser combines external guidance and self-proposals using source-level reliance weights initialized to 0.5, then provenance-aware feedback and validation changes update future source preference.Positive unified feedback shifts preference toward external experience; negative feedback shifts it toward the self proposal.

5 Experimental Evaluation

VCE-Skill is evaluated across five complementary benchmarks and four agent models, consistently improving task performance over existing skill-evolution methods. Ablations and cross-model transfer experiments further show that distilled, adaptively selected external experience drives these gains and improves skill generalization.

  • Experimental Setup: Evaluation spans five benchmarks covering question answering, document and numerical reasoning, embodied execution, spreadsheet operations, and function calling.The benchmarks are SearchQA, OfficeQA, ALFWorld, Spreadsheet, and BFCL-v4.
  • Overall Results: 3.20–4.98 points: VCE-Skill increases average scores while improving every task score across different base skill-evolution frameworks.The comparison includes paired VCE-enhanced and non-VCE variants, showing that the benefit is not specific to one evolver.
  • Efficiency: Approximately 1.1M tokens: VCE-Skill adds about 10% token overhead relative to the best-performing baseline.The cost comprises approximately 0.5M tokens for one-time experience distillation and 0.6M tokens for roughly 20 evolution iterations per benchmark.
  • Ablation Study: Removing experience distillation causes the largest degradation, with w/o Dist falling below SkillOpt on all four models.Raw version-change records contain implementation details that may not transfer directly, whereas distillation abstracts them into reusable patterns and insights.
  • Cross-Model Transfer: 6.04, 5.25, and 5.15 points: VCE-Skill improves average scores for the three target models in cross-model transfer over SkillOpt.The transfer gain exceeds the gain over the available paired non-VCE counterparts, indicating stronger cross-model transferability.

6 Conclusion … A.4 Broader Public-History Statistics

VCE-Skill combines structured experience distilled from public skill versions with trajectory-derived proposals, while the supplementary analyses detail its protocol, taxonomy, representative updates, and broader public-history statistics. These analyses compare matched sources and show that public evolution frequently modifies instructions alongside implementation and maintenance artifacts.

  • 6 Conclusion: VCE-Skill distills public skill changes into structured prior experience and adaptively fuses them with trajectory-derived proposals to guide skill evolution.The approach combines reusable prior knowledge with task-specific execution evidence, whose coverage the pilot study finds complementary.
  • Appendix for VCE-Skill: The supplementary appendix reports the motivation study, VCE-Skill mechanism, experimental details, reproducibility information, extended analyses, and structured interfaces.It is organized around three aspects: the motivation study, the mechanism of VCE-Skill, and experimental details.
  • A.1 Matched Source Comparison: The matched study compares public version histories with trajectory-driven evolution using the same skill-change unit and component, intent, and pattern taxonomy.This protocol targets differences in the skill-update knowledge exposed by the two sources.
  • A.1 Matched Source Comparison: The benchmark-matched corpus contains 400 skill-change units from each source, enabling direct comparison of component, intent, and pattern distributions.The public pool contains 38 skills and 1,266 BFCL updates, plus 21 skills and 1,089 SearchQA updates; trajectory sources produce 1,200 updates total.
  • A.2 Skill-Change Units and Annotation Taxonomy: A skill-change unit is the maximal set of edits implementing one semantic intent through one reusable modification pattern, after excluding formatting, generated-file, and dependency-lock changes.Functionally coupled edits can span files or components, while distinct intents form separate units.
  • A.2 Skill-Change Units and Annotation Taxonomy: Each change unit records affected components, update intent, and reusable modification pattern; components are multi-label, whereas intent and pattern receive one primary label.These three dimensions provide the annotation taxonomy used in the matched comparison.
  • A.3 Representative Change Units: A representative public update coordinates user-facing guidance and executable code for service compatibility, whereas a trajectory update adds extraction rules for a SearchQA failure format.The example contrasts coordinated public-history changes with instruction updates grounded in observed execution failures.
  • A.4 Broader Public-History Statistics: In a survey of 2,146 public skills, instructions are the most frequently modified component in 11 of 12 domains, with within-domain shares of 58.9%–82.9%.The survey covers 1,904 GitHub skills and 242 ClawHub skills, including 36,719 substantive adjacent-version updates; scripts, references, and configurations also recur.

A.5 Additional Analyses of Trajectory-Driven Evolution … B.6 Base-Evolver Interfaces

The analyses show that trajectory quality and failure-mode diversity strongly shape evolution outcomes, motivating VCE-Skill’s combination of public version-change experience with task-specific trajectory evidence. Its implementation distills and selects reusable experience, fuses it with provenance-tagged self proposals through adaptive feedback, and preserves each base evolver’s native optimization loop.

  • A.5 Additional Analyses of Trajectory-Driven Evolution: Broader failure-mode coverage reduces overfitting: the train–test gap falls from 2.28 to 1.04 and mean P2F count from 12.7 to 5.9.The study defines P2F as tasks passing before evolution but failing afterward, and connects narrow trajectory evidence with iterative-evolution overfitting.
  • A.5 Additional Analyses of Trajectory-Driven Evolution; B VCE-Skill Implementation Details; B.1 Offline Experience-Bank Construction: VCE-Skill uses public histories for recurring maintenance and implementation strategies and current trajectories for task-specific execution evidence, combining them through adaptive selection and fusion.The implementation includes an offline experience bank and online trajectory-grounded proposals that are fused with provenance preserved.
  • B.1 Offline Experience-Bank Construction; Distillation Prompt Definitions: The offline pipeline extracts atomic events from adjacent-version diffs, groups compatible events into update patterns, and generalizes them into skill- and domain-level insights.The source corpus contains 1,904 GitHub skills and 242 ClawHub skills across 12 task domains; duplicate or paraphrased events are merged only when their operation and applicability conditions agree.
  • B.2 Online Optimization Algorithm; B.6 Base-Evolver Interfaces: Online optimization changes proposal construction while retaining the base evolver’s candidate evaluation, retention, checkpointing, and stopping mechanisms.VCE-Skill inserts a proposal adapter between native proposal and skill-update stages, leaving rollout collection, diagnosis, evaluation, retention, stopping, and checkpoint logic unchanged.
  • B.3 Experience Selection and Aggregation: The selector chooses zero to five relevant, actionable, complementary experience entries, with task and skill compatibility primary and prior feedback adjusting receptiveness to external guidance.The selection budget is K = 5; positive preceding feedback increases receptiveness, while negative feedback requires stronger applicability evidence.
  • B.4 Attention Fusion and Skill Optimization: The fuser preserves useful self edits, uses external guidance to refine or constrain them, removes unsupported or contradictory edits, and records EXP-k and SELF-k provenance identifiers.Fused edits may replace, append, prepend, or delete content, and mixed provenance is recorded only when both sources materially support one edit.
  • B.5 Provenance Attribution and Adaptation Feedback: Source attribution rewards realized external edits with +1, self-proposal edits with −1, and mixed or unrealized edits with 0, while validation feedback adapts future reliance.Improving updates increase reliance on the source contributing more realized edits; non-improving updates shift reliance away from that source, while both sources remain active.

B.7 End-to-End Evolution Case Study · C Additional Experimental Results and Analyses

The case study shows how version changes are distilled into reusable update patterns, fused with trajectory-based proposals, and realized in an accepted skill update. Supplementary experiments extend evaluation across multiple benchmarks, models, baselines, and repeated paired runs under a common protocol.

  • B.7 End-to-End Evolution Case Study: Version changes are distilled into reusable update patterns by grouping compatible events by component, operation, and intent while preserving applicability boundaries.Patterns cite supplied event IDs, merge duplicate recommendations, separate differing conditions or outcomes, and remove implementation-specific details.
  • B.7 End-to-End Evolution Case Study: The illustrated update converts recurring incomplete-output incidents into activation cues, completeness checks, and recovery rules for structured-output skills.The case targets a trajectory that completes analysis but omits the required function call.
  • B.7 End-to-End Evolution Case Study: Four mixed-provenance edits covering JSON closure, tool matching, output formatting, and pre-output validation appear in the accepted Skill update.The selector retrieves three complementary experiences, while the base evolver proposes four edits.
  • B.7 End-to-End Evolution Case Study: 0.5740 hard score follows 0.5661 across 507 paired validation examples, with 17 wrong-to-right and 13 right-to-wrong transitions.A representative parallel-call example improves from zero matched calls out of two to two matched calls out of two; all realized edits have mixed provenance.
  • B.7 End-to-End Evolution Case Study: Mixed provenance yields f_S_t = +1 and f_t = 0, leaving the next-round attention weights balanced at λ_exp = λ_self = 0.5.The weights are reported as the resulting next-round values in the case study.
  • C Additional Experimental Results and Analyses: Supplementary experiments use SearchQA, OfficeQA, ALFWorld, Spreadsheet, and BFCL-v4 with Qwen3.5-27B, GPT-5.2, DeepSeek-v3.2, and Claude Sonnet 5.Compared methods include No Skill, LLM Skill, EvoSkill, SkillClaw, SkillOpt, and VCE-enhanced variants of three iterative methods.
  • C Additional Experimental Results and Analyses: Each supplementary configuration runs independently three times, with paired comparisons using identical benchmark instances, initial skill, evolution budget, and target model.The supplementary experiments follow the main-paper protocol unless otherwise specified.
  • C Additional Experimental Results and Analyses: GPT-5.5 performs LLM-based annotation and motivation-study analysis, while experience distillation and fusion use the corresponding target model in each configuration.All models are accessed through APIs, and experiments run on a server with an Intel Core i7-10700 CPU, NVIDIA TITAN RTX GPU, and 32 GB of RAM.

C.1 Ablation Results Across Base Self-Evolvers

Across EvoSkill, SkillClaw, and SkillOpt, VCE-Skill’s experience distillation and adaptive experience attention each contribute to stronger self-evolution performance. The complete method achieves the highest average score across all twelve base-evolver–model combinations.

  • Experimental setup: The ablation evaluates experience distillation and adaptive experience attention with EvoSkill, SkillClaw, and SkillOpt across five benchmarks and four agent models.The w/o Dist variant uses raw version-change records, whereas w/o Atte replaces adaptive attention with fixed equal weights.
  • Results: The complete VCE-Skill achieves the highest average score for every base self-evolver and agent model across all twelve combinations.Removing either component reduces performance relative to the complete method.
  • Results: Removing experience distillation causes the larger degradation in most settings, indicating that distilled update patterns and evolution insights transfer more effectively than raw version-change records.Removing adaptive experience attention also consistently reduces performance, although fixed equal weighting is less damaging in most settings.
  • Conclusion: Experience distillation and adaptive experience attention provide complementary improvements that generalize across three different self-evolution frameworks.Distillation converts concrete version changes into reusable evolution knowledge, while adaptive attention controls external experience during self-evolution.

C.2 Hyperparameter and Attention Sensitivity

Sensitivity experiments across five benchmarks support K = 5, η = 0.1, and clipping bounds of [0.3, 0.7] as VCE-Skill’s default configuration. The attention step size is most influential, while moderate selection budgets and constrained clipping maintain stability.

  • Experimental setup: The experiments fix source weights at 0.5/0.5 and vary the experience-selection budget K, attention step size η, or clipping bounds independently.Table 5 reports mean performance on each benchmark.
  • Results: K = 5 achieves the highest score on every benchmark, while K = 4 remains within 0.02–0.11 points and K = 7 reduces performance by 1.11–1.88 points.Experiments use SkillOpt with Qwen3.5-27B and vary one hyperparameter at a time.
  • Results: η = 0.1 consistently performs best, whereas η = 0.2 decreases performance by 2.28–3.88 points across the five benchmarks.The attention step size has the strongest effect because it controls the rate of source-weight adaptation.
  • Results: The default clipping bounds of [0.3, 0.7] are comparatively stable across the five benchmarks.Moderately constrained bounds maintain stable attention updates, whereas overly permissive bounds reduce performance.

C.3 Complete Cross-Model Transfer Results

VCE-Skill improves cross-model transfer across all evaluated source–target pairs and benchmarks, without target-side evolution. The gains are consistently positive, averaging 5.56 points overall.

  • Conclusion: The consistent gains across every source model, target model, and benchmark indicate stronger transferable guidance from VCE-Skill.Transferred skills receive no target-side evolution, so the observed improvements reflect the guidance produced during source-side evolution.
  • Experimental setup: The transfer evaluation applies each evolved skill directly to three other target models, producing 12 directed source–target pairs across five benchmarks.The average gain compares SkillOpt+VCE-Skill against SkillOpt for each source–target transfer.
  • Results: SkillOpt+VCE-Skill outperforms SkillOpt in all 60 benchmark-level comparisons across 12 directed source–target pairs.The evaluation covers four models, five benchmarks, and every ordered source–target pairing without further target-side evolution.
  • Results: 5.56 points is the overall mean improvement, with average gains ranging from 5.15 to 6.19 points.Source-model averages are 5.48 for Qwen3.5-27B, 5.37 for GPT-5.2, 5.71 for DeepSeek-v3.2, and 5.69 for Claude Sonnet 5.

D Human Audit of LLM-based Decisions

A human audit evaluates six LLM-based semantic decision types used in the motivation study and VCE-Skill. Across these decisions, high acceptance rates and consistent annotator agreement support their reliability, while not implying error-free judgments.

  • Audit Scope and Sampling: The audit covers Motivation-study Taxonomy, Skill-Change Coding, Change Abstraction, Experience Generation, Experience Selection, and Source Attribution.Each decision type is evaluated using sampled experimental records and human annotation.
  • Annotation Process: Two annotators independently assign Accept or Reject labels under anonymized conditions, with disagreements resolved by a third annotator.A 20-instance pilot for each decision type is repeated when agreement falls below κ = 0.7.
  • Audit Results and Summary: 86.5%–100.0% HAR and 0.75–0.84 Cohen’s κ summarize the six audited decision types.HAR uses final adjudicated labels, whereas κ measures agreement between the two initial annotators before adjudication.
  • Audit Results and Summary: 94.0% HAR and 0.84 κ characterize Motivation-study Taxonomy, while Skill-Change Coding reaches 100.0% HAR and 0.84 κ.All audited Skill-Change Coding instances are accepted after adjudication.
  • Audit Results and Summary: Change Abstraction, Experience Generation, and Source Attribution each exceed 90% HAR, whereas Experience Selection records the lowest HAR at 86.5%.The results indicate that most generated events, hierarchical experience entries, and provenance assignments align with supporting changes; selection is more context-dependent.
  • Audit Results and Summary: Overall, the high acceptance rates and consistent agreement support the reliability of the audited decisions for analysis and optimization, but do not establish error-free judgments.The conclusion applies to outputs assessed under the defined audit criteria.
Loading 2608.16544v1…