PaperlayerView exact revision on arXiv

Source-linked AI summary

WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution

Liyan Tang, Cyrus Rashtchian, Chun-Sung Ferng, Andrew Tomkins, Da-Cheng Juan, Tu Vu

arXiv:2608.27454v1cs.AIcs.CL

TL;DR

Agent-skill development lacks a persistent way to organize experience for reuse across iterations. WikiSkill co-evolves skills with a persistent wiki, outperforming existing methods across benchmarks while enabling transfer across models and model families.

  • Problem

    Agent-skill development remains difficult because experience and lessons are not maintained as a separate, evolving knowledge representation for systematic reuse.

  • Method

    WikiSkill separates raw execution traces, persistent structured knowledge, and evolving skills in a continual loop that consolidates experience and gates skill updates.

  • Results

    Across five benchmarks and five models, WikiSkill consistently outperforms existing skill-evolution methods; within Qwen, average gains are 12.3%, 17.5%, and 23.9% for 4B, 9B, and 27B models.

  • Takeaways & Limitations

    Persistent knowledge accumulation supports effective skill evolution, while evolved skills can transfer across models and sometimes outperform self-evolved skills.

  • Takeaways & Limitations

    The study does not evaluate skill retrieval or triggering because active skills are directly injected into prompts.

Abstract

from arXiv · show

Agent skills package specialized knowledge and workflows into reusable resources that extend AI agent capabilities. Recent work automatically discovers such skills from agent experience, which enables agents to progressively adapt through interaction. However, the insights that guide skill development typically remain scattered across optimization histories, limiting their systematic reuse across iterations. We introduce WikiSkill, a framework that co-evolves agent skills with a persistent knowledge base (wiki). At a high level, WikiSkill separates raw execution experience, accumulated knowledge, and executable skills, while continuously consolidating experience into the wiki, which subsequent skill updates can build on. Across diverse benchmarks and models, WikiSkill consistently outperforms state-of-the-art skill-evolution methods and improves over no-skill baselines in most model-benchmark settings. We find that skill evolution complements model scaling: larger models generally benefit more from evolved skills, while smaller models with skills can outperform substantially larger models without them. We also find that evolved skills transfer effectively across models and model families, and skills evolved by other models can outperform self-evolved skills. Finally, our ablation studies confirm that persistent knowledge accumulation in the wiki is critical for effective skill evolution. These results demonstrate the benefits of systematically accumulating and refining agent experience for developing reusable and transferable skills.

1. Introduction

WikiSkill addresses the challenge of evolving reusable agent skills by persistently organizing execution experience into a structured knowledge base that guides skill updates. Across five benchmarks and five models, it outperforms existing skill-evolution methods, improves over no-skill settings in most cases, and enables scalable and transferable skill gains.

  • 12.3%, 17.5%, and 23.9% average improvements for Qwen 4B, 9B, and 27B models show that skill gains increase with model scale.The reported gains correspond to the 4B, 9B, and 27B models, respectively.
  • Qwen-3.5-9B with WikiSkill outperforms Qwen-3.6-27B without skills, achieving 47.4% versus 39.4%, respectively.This demonstrates that evolved skills can compensate for substantial model-scale differences.
  • Evolved skills transfer across model families and can outperform self-evolved skills, with Qwen-3.5-9B reaching 70.2% versus 63.4% on ALFWorld.The 70.2% result uses a Qwen-3.6-27B-evolved skill, whereas 63.4% uses Qwen-3.5-9B’s own skill.
  • WikiSkill consistently outperforms existing skill-evolution methods across five benchmarks and five models, while improving over no skills in most settings.The evaluation spans mathematical reasoning, web search, spreadsheet manipulation, long-context document question answering, and interactive embodied tasks, using Qwen, Gemma, and Gemini models.
  • WikiSkill co-evolves agent skills with a persistent knowledge base that continually organizes and refines knowledge from agent experience.Its workspace separates immutable execution traces, structured wiki knowledge, and evolving procedural skills, with inference, maintenance, proposal, and gating components supporting each iteration.
  • Persistent knowledge accumulation is critical for effective skill evolution, as confirmed by the framework’s ablation studies.The contribution summary identifies persistent knowledge accumulation as an important factor in WikiSkill’s performance.

2. Problem Setup

The problem setup formalizes iterative skill evolution for tool-using LLM agents over train, validation, and test task splits. WikiSkill jointly evolves active procedural skills and a persistent knowledge base, using training rollouts and validation gating to improve unseen-task performance.

  • 2. Problem Setup: Tasks are partitioned into disjoint training, validation, and testing splits, with performance defined as average domain-specific prediction scores over each split.Each task pairs an instance x_i with a ground-truth answer y_i, and correctness is measured by f(ŷ_i, y_i) ∈ [0, 1].
  • 2. Problem Setup: An agent is an LLM-based system with tools and modular filesystem-based skills that guide multi-step execution trajectories and final answer generation.Skills package procedural knowledge into instructions, scripts, and other resources, while trajectories contain observations and actions that may call available tools.
  • 2. Problem Setup: WikiSkill represents each iteration by the joint state (S_k, W_k), combining active procedural skills with a persistent knowledge base that compounds across iterations.Candidate skill updates can be validation-gated and rolled back after score degradation, whereas the knowledge base persists.
  • 2. Problem Setup: Starting from empty skills and knowledge, WikiSkill co-evolves both state components across iterations using training rollouts, pattern consolidation, and validation gating to maximize test performance on unseen tasks.The framework separates immutable execution traces, persistent wiki knowledge, and active procedural instructions into distinct layers.

3. Methodology

WikiSkill co-evolves executable agent skills with a persistent wiki through a three-layer architecture and an iterative loop of rollout analysis, skill proposal, and validation gating. Immutable traces are consolidated into historical knowledge that guides subsequent skill refinement while active skills are injected into the inference agent.

  • 3.1 Three-layer knowledge architecture: WikiSkill separates raw execution traces, compounding wiki knowledge, and active evolved skills into three distinct workspace layers.The raw layer preserves complete step-by-step interactions immutably; the wiki layer records patterns, historical evolution, and skill impacts; the skills layer stores executable SKILL.md files and their motivating PURPOSE.md mappings.
  • 3.2 Evolution loop: Each iteration runs the inference agent with active skills, consolidates rollout traces into the wiki, proposes a skill update, and filters it through validation.Training rollouts cannot access the wiki; the Wiki Maintainer analyzes traces and existing knowledge, while the Skill Proposer uses the resulting resources to formulate changes.
  • Wiki Maintainer: The Wiki Maintainer samples successful and failing traces, performs root-cause analysis and strategy extraction, and incrementally updates or creates pattern pages.It receives the full prior wiki alongside sampled traces, using patch-based edits to refine persistent evidence and solutions.
  • Skill Proposer: The Skill Proposer autonomously inspects selected wiki patterns and raw traces before creating a new skill or applying an atomic patch to one existing skill.Its ReAct-style process uses the wiki index, skill-impact history, and summarized training outcomes to avoid exhausting the context window.
  • Validation gating: Candidate skills are accepted only when validation improves the tracked best score, with rejected modifications discarded and accepted transitions recorded for future proposals.The system initializes the best score from the empty-skill baseline, can terminate when validation reaches 1.0, and maintains an audit trail of each intervention.

4. Experiments and Results

Across five diverse benchmarks and five models, WikiSkill consistently outperforms competing skill-evolution methods and no-skill baselines. Its benefits grow with model capability, transfer across models, and depend on task characteristics and execution ability.

  • Experimental setup: Experiments cover mathematical reasoning, web search, spreadsheet manipulation, long-context document QA, and interactive embodied tasks across closed and open-weight models.Each method is evaluated over three independent evolution runs, with scores averaged across the resulting skill sets and significance tested by paired bootstrap at p<0.05.
  • WikiSkill yields consistent improvements across models and datasets: WikiSkill achieves the highest average performance across all five models, improving over the strongest competing method by 3.3–12.0 points and over no-skill baselines in most model-dataset pairs.The gains include Gemini-3.5-Flash improvements from 33.0% to 72.6% on LiveMath and from 50.5% to 76.6% on SpreadSheet, plus Qwen-3.6-27B from 52.8% to 77.6% on ALFWorld.
  • The benefits of skill evolution increase with model capability and complement model scaling: Within the Qwen family, WikiSkill’s average improvement increases with scale, from +12.3 points for 4B to +23.9 points for 27B, while skills can offset model-size differences.Qwen-3.5-9B with WikiSkill reaches 47.4% average accuracy versus 39.4% for Qwen-3.6-27B without skills.
  • The benefits of skill evolution also vary substantially across datasets: Skill benefits vary by dataset: Qwen-3.6-27B gains 24.8 points on ALFWorld but only 11.6 on OfficeQA, while smaller models can struggle with long-context search workflows.The same analysis reports 14.1 points on SealQA for Qwen-3.6-27B and slight degradation for Qwen-3.5-4B in long-context settings.
  • The transferability of evolved skills depends on whether they capture general procedures or model-specific workarounds: Transferability depends on whether skills encode general procedures or model-specific workarounds: LiveMath skills transfer strongly, whereas some SpreadSheet skills produce negative transfer.Qwen-3.5-4B and Qwen-3.6-27B skills improve Gemini-3.5-Flash from 33.0% to 67.5% and 73.9%, respectively.
  • Evolved skills transfer effectively across models, and transferred skills can outperform self-evolved skills: WikiSkill skills frequently transfer across models and can outperform self-evolved skills, such as Qwen-3.6-27B skills reaching 50.5% on Qwen-3.5-9B SpreadSheet versus 33.6% self-evolved.The no-skill baseline in that comparison is 24.3%.

5. Analysis and Discussion

WikiSkill’s persistent wiki substantially improves skill evolution, while exposing the inference agent to the wiki during rollouts can reduce final skill quality. The system continually accumulates model- and benchmark-dependent knowledge patterns that support ongoing skill refinement through recorded proposals, rejections, and evidence.

  • Persistent wiki knowledge dramatically improves skill evolution: 15.0 percentage points: Persistent wiki access for the Skill Proposer raises average performance from 48.7% to 63.7%, including LiveMath gains from 51.3% to 72.6% and SpreadsheetBench gains from 49.9% to 76.6%.This comparison is reported for Gemini-3.5-Flash with wiki access disabled for the Inference Agent; without persistent accumulation, the Skill Proposer struggles with intricate failure modes.
  • Wiki access for the Inference Agent during evolution degrades final skill quality: 2.8 percentage points: Giving the Inference Agent wiki access during training rollouts lowers average performance from 63.7% to 60.9% and LiveMath performance from 72.6% to 64.8%.The authors hypothesize that direct access to both skills and wiki knowledge during rollouts may provide task-solving knowledge without improving the evolved skill.
  • WikiSkill continuously accumulates wiki patterns while producing concise skills: Wiki pattern accumulation and skill structure vary across models and benchmarks: Qwen models produce 118.9–128.6-line skills, while Gemma-4-31B and Gemini-3.5-Flash produce 45.1- and 81.2-line skills.Across benchmarks, SpreadSheet produces the longest skills at 142.5 lines and most wiki patterns at 9.8, whereas LiveMath produces the shortest skills at 84.6 lines and fewest patterns at 4.4.
  • Skill refinement continues throughout the evolution process: Skill refinement continues beyond the initial stage: accepted updates comprise 39%–52% of updates in iterations 0–1 across models and 39%–58% across benchmarks.Accepted updates are also grouped into middle iterations 2–4 and late iterations 5–7, indicating continued evolution after initialization.

6. Related Work

Prior work studies reusable agent skills and experience-driven self-improvement, while skill-augmented agents also investigate retrieving relevant skills during execution. WikiSkill instead emphasizes persistent consolidation of experience into structured knowledge so later skill updates can build systematically on it.

7. Conclusion

WikiSkill co-evolves agent skills with a persistent, compounding knowledge base that supports increasingly integrated knowledge across iterations. Its orchestrated loop consolidates experience, proposes refinements, and gates changes using validation performance, while outperforming existing skill-evolution methods across five benchmarks and five inference models.

  • WikiSkill co-evolves agent skills with a persistent, compounding knowledge base (wiki).
  • Its three-layer workspace enables skill development to build on increasingly well-supported and integrated knowledge across iterations.
  • The orchestrated loop consolidates experience into the wiki, proposes skill refinements from accumulated knowledge, and gates changes using validation performance.
  • WikiSkill consistently outperforms existing skill-evolution methods across five benchmarks and five inference models.

Limitations

WikiSkill’s evaluation does not assess skill retrieval or triggering because active skills are directly injected into prompts. Its strict validation gate also excludes neutral proposals that might enable later improvements.

  • Directly injecting active skills into prompts leaves skill retrieval and triggering unevaluated, despite their growing importance as available skills increase.This design follows prior work and aims to isolate skill quality by avoiding confounding effects from retrieval.
  • Requiring every accepted proposal to improve validation scores excludes neutral proposals that preserve immediate performance but could support gains in later iterations.The study uses this strict validation criterion as its gating rule.

A. Method Details · A.1. Algorithm · A.2. Distribution of Accepted Skill Updates

WikiSkill iteratively evolves skills by rolling out on training tasks, consolidating traces into a wiki, proposing modifications, and accepting or rolling back updates through validation gating. The method retains the wiki after rejected skill proposals and separately analyzes when updated proposals are accepted.

  • A.1. Algorithm: WikiSkill’s evolution loop combines inference, wiki maintenance, proposal generation, validation gating, and final wiki updates at each iteration.The loop uses active skills on training tasks, consolidates sampled traces into an intermediate wiki, generates candidate modifications, and records proposal outcomes and skill diffs.
  • A.1. Algorithm: Validation gating either accepts the proposed skill update or rolls back to the previous skill state.The algorithm explicitly distinguishes the proposed S′_k from the retained S_k−1 when an update is rejected.
  • A.1. Algorithm: The algorithm requires training tasks, validation tasks, a performance metric, and a fixed number of iterations.These inputs define the training and validation workflow and the performance criterion used during skill evolution.
  • A.1. Algorithm: Before evolution, WikiSkill establishes a baseline validation performance from rollouts using the initial skill set.The baseline is computed from validation-task traces sampled under S0 and stored as Rbest.
  • A.1. Algorithm: At each iteration, the inference agent rolls out on training tasks with the current skills and samples a subset of resulting traces for wiki maintenance.Training rollouts use S_k−1, and the maintainer processes a sampled subset rather than necessarily all traces.
  • A.1. Algorithm: When a proposal is rejected, WikiSkill rolls back the skills but retains the accumulated wiki.The rejection step sets S_k to S_k−1 while preserving the wiki state for subsequent evolution.
  • A.2. Distribution of Accepted Skill Updates: The paper reports the distribution of conditions under which updated skill proposals are accepted in Table 5.The supplied passage identifies Table 5 as the source for acceptance behavior but does not provide its cell values.

B. Dataset Details and Splits

The evaluation uses five benchmarks with distinct reasoning, search, long-context, and interactive-task demands, while matching prior work’s data splits and tool configurations. Because validation sets are small, reported scores average three independent evolutionary-pipeline runs and use paired bootstrap significance testing.

  • Benchmark coverage: Five benchmarks assess diverse capabilities, including complex mathematical reasoning, scholarly web-based question answering, long-context financial evidence synthesis, and interactive task completion.LiveMath tests quantifiers and extremal conditions; SealQA tests search-query formulation and answer extraction; Treasury-bulletin tasks require multi-page evidence synthesis; ALFWorld is interactive.
  • Dataset splits and tools: Table 6 reports benchmark sample counts across training, validation, and test splits, together with interaction modes and available tools.The splits and toolsets are strictly matched with prior work (Alzubi et al., 2026; Yang et al., 2026).
  • Interaction modes: LiveMath is single-step and tool-free, whereas SealQA provides web search; Treasury tasks provide oracle reference pages while retaining local text-processing tools.The Treasury setup follows Yang et al. (2026), using pre-parsed oracle pages as initial evidence and glob, grep, and read for document search and inspection.
  • Evaluation robustness: Small validation splits can make gating decisions noisy, so reported scores average test performance over three independent runs and apply paired bootstrap significance testing.The robustness procedure follows established setups from Alzubi et al. (2026) and Yang et al. (2026).

C. Implementation Details · D. Baseline Details and Optimizer API Call Analysis · D.1. Baseline Methods

The implementation uses stratified execution-trace sampling and bootstrap-based significance testing with explicit rules for declaring top performers. Baselines represent distinct trace-analysis, candidate-frontier, and ReflACT-based skill-optimization pipelines.

  • C. Implementation Details: Up to 8 traces are sampled per iteration, stratified between failing traces for root-cause analysis and passing traces for effective-strategy discovery.The allocation permits up to 5 failing and 3 passing traces.
  • C. Implementation Details: Execution logs are capped at 15,000 characters before being injected into the prompt.
  • C. Implementation Details: Paired bootstrap tests use 1,000 iterations per benchmark, resampling test instances with replacement to compute candidate accuracies and pairwise performance margins.
  • C. Implementation Details: A method is declared the sole top performer only when its observed or macro-average gain over every competitor is significant at p<0.05.Otherwise, statistically indistinguishable top-ranked methods are not assigned a unique top designation.
  • D.1. Baseline Methods: Trace2Skill (Ni et al., 2026) analyzes passing and failing training traces in parallel, then hierarchically merges structured patches into an applied patch set.
  • D.1. Baseline Methods: EvoSkill (Alzubi et al., 2026) searches a bounded frontier of candidate skill programs using round-robin training categories, failure-only feedback, validation scoring, and proposal histories.
  • D.1. Baseline Methods: SkillOpt (Yang et al., 2026) uses a six-stage ReflACT pipeline to reflect on full traces, hierarchically aggregate and select patches, and validate updates to one monolithic skill.

D.2. Optimizer API Call Complexity · 3. Hierarchical reduce & apply stage: The 𝑁train · E. System and Agent Prompts

WikiSkill achieves O(1) optimizer API-call complexity with respect to training-set size under full-batch evolution, requiring 1 + T_ReAct calls per iteration, while prior methods scale with the number of minibatches or trajectories. The appendix also specifies task-specific inference prompts and detailed Wiki Maintainer and Skill Proposer workflows.

  • D.2. Optimizer API Call Complexity: 1 + T_ReAct optimizer LLM calls are required per WikiSkill iteration regardless of training instances, yielding O(1) complexity with respect to N_train.WikiSkill uses full-batch processing, B = N_train, and T_ReAct is roughly 10–20 in the experiments.
  • D.2. Optimizer API Call Complexity: WikiSkill’s constant call complexity may increase inference cost on some datasets but accompanies consistent performance gains over prior skill-evolution methods.The comparison is made across the evaluation settings described for the optimizer-complexity analysis.
  • D.2. Optimizer API Call Complexity: EvoSkill and SkillOpt scale as O(N_train/B), whereas Trace2Skill is lower-bounded by O(N_train) because it analyzes every training trajectory individually.SkillOpt uses approximately 6–8 optimizer calls per minibatch step, while Trace2Skill additionally performs hierarchical patch merging and final formatting.
  • 3. Hierarchical reduce & apply stage: The N_train: WikiSkill’s full-batch setup lets the Skill Proposer dynamically search, select, and read execution traces on demand, unlike EvoSkill and SkillOpt’s best-performing minibatch settings.The full-batch configuration processes the entire training set at once per iteration.
  • E. System and Agent Prompts: The appendix provides exact system prompts for inference agents across tasks, the Wiki Maintainer, and the WikiSkill Skill Proposer.The task prompts cover mathematical reasoning, factual web question answering, spreadsheet manipulation, OfficeQA document work, and ALFWorld interaction.
  • Wiki Maintainer Agent System Prompt: The Wiki Maintainer is instructed to analyze execution traces deeply, document success and failure patterns with root causes and commands, and maintain structured incremental wiki files.Its required output updates patterns, the index, and the evolution log, while index entries must concisely state the problem, root cause, and fix.
  • Skill Proposer Agent System Prompt: The Skill Proposer must read the wiki and prior skill impacts before examining targeted traces, then create, patch, or decline a skill change using structured proposals.The workflow requires reading at least four execution traces before proposing a change and favors concise, actionable patches to partially correct skills.
Loading 2608.27454v1…