PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution

Hongyi Liu, Haoyan Yang, Tao Jiang, Bo Tang, Feiyu Xiong, Yuyu Luo, Zhiyu Li

arXiv:2605.18401v2cs.CLcs.AI

TL;DR

Long-horizon agents need a way to govern the reuse of noisy trajectories as skill libraries grow. SkillsVote selects skills before execution and gates library updates using outcome attribution; across six benchmark/backbone pairs, it improves overall scores over no-skill baselines and is best or tied-best online.

  • Problem

    Long-horizon agents lack a durable, governable way to reuse noisy, environment-bound trajectories, while skill libraries scale with redundancy and uneven quality.

    Source: §1 · p. 2

  • Method

    SkillsVote governs Agent Skills across the lifecycle by selecting task-relevant skills before execution and attributing outcomes to gate post-task library updates.

    Source: §1 · p. 3, §3 · p. 4

  • Results

    Across six benchmark/backbone pairs, SkillsVote beats no-skill baselines and is best or tied-best online; Terminal-Bench 2.0 GPT-5.2 avg@5 Accuracy rises ↑2.6 pp.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The lifecycle framing treats pre-task distraction and post-task library pollution as coupled risks in growing skill libraries.

    Source: §5 · p. 10

Abstract

from arXiv · show

Long-horizon LLM agents generate traces that could become reusable experience, but raw trajectories are noisy, local, and hard to govern. Agent Skills offer a structured artifact for combining procedural guidance, executable resources, and applicability boundaries. Yet open skill ecosystems contain redundant, uneven, environment-sensitive artifacts, and indiscriminate updates can pollute future context. We present SkillsVote, a lifecycle-governance framework for Agent Skills across collection, recommendation, attribution, and evolution. SkillsVote profiles a million-scale open source corpus for environment requirements, quality, and verifiability, and synthesizes tasks for verifiable skills. Before execution, it performs agentic library search over structured skill folders to expose instructional context. After execution, it decomposes trajectories into skill-linked subtasks, attributes outcomes to skill-guided execution, agent exploration, environment, and result signals, and admits only successful reusable discoveries to evidence-gated updates. Experiments on Terminal-Bench 2.0 and SWE-Bench Pro show that SkillsVote improves agent performance on challenging agentic coding benchmarks. The gains arise from two complementary pathways: online evolution over task streams at test time and offline transfer via frozen libraries built from either historical trajectories or curated open source skills.

1 Introduction

SkillsVote governs Agent Skills across collection, recommendation, attribution, and evolution, using a profiled open-source corpus and a closed loop that controls which experiences enter the library. It aims to make skill reuse more reliable while limiting pollution from weakly supported updates.

1 Introduction

  • Raw trajectories are noisy, environment-bound, and poor for durable reuse, while Agent Skills package instructions, executable resources, dependencies, and applicability conditions into auditable artifacts.This structured form is more compact than full trajectories while retaining more executable context than isolated natural-language summaries.Source: §1 · p. 2
  • SkillsVote profiles more than one million open-source skills amid ecosystem-scale redundancy, uneven quality, safety risks, and library pollution from indiscriminate updates.The paper frames collection, governance, profiling, recommendation, evaluation, and evolution as coupled processes.Source: §1 · pp. 2–3
  • Before execution, SkillsVote searches a structured skill library to select a small, relevant, low-redundancy set; afterward, it attributes outcomes across skills, agent exploration, environment, and evaluation signals.The figure depicts this pre-task recommendation and post-task attribution loop, in which skill-linked subtasks inform potential edits or new skills.Source: Figure 1 · p. 2, §1 · p. 3
  • Attribution gates library evolution, admitting reusable discoveries while preventing spurious successes or environment- and evaluation-caused failures from driving irrelevant edits.This links recommendation and controlled evolution in an auditable lifecycle framework and reduces the risk of indiscriminate updates.Source: §1 · p. 3
  • SkillsVote improves online evolution, offline transfer, and recommendation-controlled skill use on Terminal-Bench 2.0 and SWE-Bench Pro.The evaluation examines recommendation versus direct exposure to an initial library, offline transfer from historical trajectories, and online evolution across a test-time task stream.Source: §1 · p. 3

2 Related Work

Related work traces agent experience learning from memories and strategies toward executable skills, while emphasizing ecosystem governance, skill selection, and skill-library evolution. SkillsVote focuses on auditable, portable skill artifacts and adds subtask-level attribution to evidence-gated library updates.

Evolution of Agent Experience Learning

  • SkillsVote treats skills as procedural text, scripts, dependencies, and applicability boundaries, making experience auditable, versionable, and portable across harnesses.Earlier approaches include unstructured memories [62] [77] [80], workflows and SOPs [13] [65], strategies [5] [6] [7] [47] [61] [72] [76], and callable tools or interfaces [18] [30] [33]; broader systems frame memories, rules, skills, protocols, and harness components as deployment-time artifacts [25] [27] [28] [73] [74] [79].Source: §2 · p. 3

Agent Skill Ecosystems, Retrieval and Evaluation

  • SkillsVote profiles open-source skills for format, dependencies, quality, and verifiability, then uses filesystem-native inspection to produce compact guidance for combining selected skills.Existing evaluations stress utility, compositional use, portability, security, dependencies, and harness compatibility [23] [8] [48]; SkillRouter routes over full skill bodies [78], while DCI interacts directly with source documents [24].Source: §2 · pp. 3–4

Skill-Centric Agent Self Evolution

  • SkillsVote factorizes trajectories into judged, skill-linked subtasks, localizes used knowledge and outcome responsibility, and admits only reusable successful exploration into library evolution.Prior systems convert coarse session- or trajectory-level evidence, often with verifier or environment feedback, into skill artifacts [2] [14] [35] [37] [38] [53] [58] [69] [70] [71] [75] [81].Source: §2 · p. 4
  • Skill-library evolution research either trains policies to retrieve, use, and distill or revise skills [34] [46] [51] [60] [64] [66] [67], or converts coarse experience into reusable artifacts.As skills become installable and shareable file artifacts, ecosystem governance and correct selection, composition, and use remain central concerns.Source: §2 · pp. 3–4

3 Approach

SkillsVote governs Agent Skills across the lifecycle by controlling which skills enter the solver’s context and what execution evidence can update the library. It recommends task-relevant skills, distills trajectories into attributed reusable units, and applies conservative evidence-gated updates.

3 Approach

  • SkillsVote treats skills as lifecycle artifacts, controlling which enter the solver context and which execution evidence may update the library.Corpus collection, profiling, and task synthesis from skills are described as preprocessing details in Appendix D.1.Source: §3 · p. 4

3.1 Skill Recommendation via Agentic Library Search

  • A separate recommendation agent searches skill folders, reads candidate guidance and resources, and exposes a compact set of environment-fitting, complementary skills with a usage guide.The recommendation record also supports later attribution by recording whether exposed skills were used and contributed reusable discoveries.Source: §3.1 · p. 4

3.2 Distilling Execution Traces into Evolvable Units

  • SkillsVote divides trajectories into semantically complete subtasks at changes in objective, evaluation signal, or skill context, then attributes outcomes, responsibility, and reusable skill-related discoveries.Each subtask has one standalone objective, one primary evaluation signal, and at most one associated skill context; attribution distinguishes environment, human, or absent signals, skill-guided execution or exploration, and reusable deltas from task-specific or repetitive details.Source: §3.2 · pp. 5–6

3.3 Evidence-Based Controlled Skill Evolution

  • SkillsVote permits updates only from successful subtasks with reusable exploration, aggregates evidence for the same change, and routes it to a minimal skill edit, a new skill, or no update.Changes must be supported by attributed evidence, localized to the relevant skill boundary, and expressed as reusable procedural knowledge rather than a trajectory recap.Source: §3.3 · p. 6

4 Experiments

The experiments are organized around three lifecycle questions, including whether frozen skill libraries can improve performance on unseen tasks.

4.1 Experimental Setup

  • The evaluation is organized around three lifecycle questions.Source: §4.1 · p. 6
  • One question tests whether frozen libraries built from historical trajectories or curated open-source skills can improve unseen tasks.Source: §4.1 · p. 6

2. Online skill evolution. Can SkillsVote accumulate useful skills over a sequential task stream?

SkillsVote improves performance across the evaluated benchmark-backbone pairs, while its analyses show how recommendation filters harmful skill exposure and offline evolution builds transferable procedures. The results cover both online updates over a task stream and offline reuse of frozen skill libraries.

2. Online skill evolution. Can SkillsVote accumulate useful skills over a sequential task stream?

  • Evaluation compares online evolution from an empty library with offline frozen-library recommendation on Terminal-Bench 2.0 and SWE-Bench Pro public.The reported metrics are avg@5 Accuracy and avg@1 Resolve Rate, respectively. Configurations include GPT-5.2, GPT-5.4 mini, and GPT-5.5; offline libraries include historical TB-Pro and 10K curated open-source skills.Source: §2 · pp. 6–7

4.2 Main Results

  • Across six benchmark-backbone pairs, SkillsVote improves over no skills and is best or tied-best online; Terminal-Bench 2.0 avg@5 Accuracy rises by ↑2.6 pp (GPT-5.2), ↑1.1 pp (GPT-5.4 mini), and ↑0.9 pp (GPT-5.5).Tables report the main results on Terminal-Bench 2.0.Source: Table 1 · p. 7, §4.2 · p. 7
  • On SWE-Bench Pro public, SkillsVote improves avg@1 Resolve Rate by ↑2.7 pp (GPT-5.2), ↑2.6 pp (GPT-5.4 mini), and ↑1.2 pp (GPT-5.5).The passage reports these gains as part of the overall improvement across all six benchmark-backbone pairs.Source: §4.2 · p. 7, Table 2 · p. 8

4.3.1 Routing over Large Skill Libraries

  • At full scale, GPT-5.5 reaches 62.7 FC@10 versus SkillRouter’s 46.7; GPT-5.4 matches SkillRouter on Hit@1 and reaches 54.7 FC@10.FC@10 declines for all methods as the pool grows, while average API cost changes little; GPT-5.4 mini drops at 50k and full scale.Source: Table 3 · p. 8, §4.3.1 · p. 9, Figure 4 · p. 9

4.3.2 Recommendation Controls Negative Transfer

  • On Terminal-Bench 2.0 Hard, recommendation shifts online task-level mean gain/loss from +3.3/−6.7 to +6.0/−6.0; offline, positive contribution rises from +11.3 to +15.3 and loss falls from −3.3 to −2.0.Recommendation filters sparse, underspecified, or weakly related skills from the solver context, complementing evolution by deciding whether skills should be exposed to a task.Source: §4.3.2 · p. 9, Figure 5 · p. 9

4.3.3 Offline Evolution Accumulates Transferable Procedures

  • Across checkpoints, frozen libraries transfer increasingly well to unseen Terminal-Bench 2.0 Hard tasks even as Terminal-Bench Pro performance fluctuates; library growth comes through skill creation and editing.Post-task attribution uses source benchmark feedback only after completion, and reusable exploration excludes benchmark-specific constants and gold outputs.Source: §4.3.3 · p. 10, Figure 6 · p. 10

5 Conclusion · Appendix · A Extended Analysis

SkillsVote governs Agent Skills as persistent lifecycle artifacts, linking task-conditioned recommendation and execution-readiness profiling with outcome attribution and evidence-gated evolution. On Terminal-Bench 2.0 and SWE-Bench Pro, governed libraries improve frozen agents through online evolution and recommendation from frozen libraries.

5 Conclusion

  • On Terminal-Bench 2.0 and SWE-Bench Pro, governed skill libraries improve frozen agents through online evolution and recommendation from frozen libraries.The frozen libraries are built from historical execution trajectories or curated open source skills.Source: §5 · p. 10
  • SkillsVote connects a million-scale open source skills corpus with readiness profiling, task-conditioned recommendation, subtask-level outcome attribution, and evidence-gated evolution.Source: §5 · p. 10
  • The lifecycle approach addresses two risks: irrelevant skills can distract agents before execution, while weakly supported or misattributed experience can pollute libraries afterward.Source: §5 · p. 10
  • Before a task, SkillsVote searches structured skill folders to recommend skills conditioned on the task.Source: §5 · p. 10
  • After a task, it uses subtask-level attribution to admit only successful, reusable discoveries supported by evidence into persistent skills.Source: §5 · p. 10
  • This conservative update process turns execution traces into persistent skills for long-horizon agents.Source: §5 · p. 10

A.1 Skill Characteristics

SkillsVote profiles an open-skill corpus with uneven quality and verifiability, concentrated categories and names, and substantial runtime requirements. Its findings motivate explicit governance rather than relying on popularity or treating every skill as directly task-ready.

Skill quality and verifiability

  • Consistency and orientation are common among evaluated skills, but substantially fewer satisfy completeness, success verifiability, and task constructability.Thus, many skills provide coherent reusable guidance, while fewer can directly support low-ambiguity, benchmark-constructable tasks.Source: §A.1 · p. 18, Figure 7 · p. 18

Skill categories

  • Development dominates the more than 290K governed skills, with strong Development–AgentMeta co-occurrence and visible links to DevOps and Testing.These overlaps suggest that adjacent software-engineering domains are often represented through accumulated development-oriented skills rather than standalone artifacts.Source: §A.1 · p. 18, Figure 8 · p. 19

Skill filtration

  • More than 1.68M collected skills undergo format validation, deduplication, and quality analysis, while repository stars remain concentrated below 100 throughout curation.Stricter filtering shifts only a limited share toward higher-star repositories, so popularity alone is not a reliable proxy for validity, distinctness, or execution readiness.Source: §A.1 · p. 18, Figure 9 · p. 19

Skill names

  • Repeated skill names are highly concentrated: skill-creator appears more than 5K times and frontend-design more than 3K times.Beyond meta-skills, repeated names mostly describe broad workflows such as design, debugging, testing, and document processing.Source: §A.1 · p. 19

Skill runtime requirements

  • Across approximately one million evaluated skills, Linux and macOS dominate runtime profiles, and workspace or system write scopes are more common than read-only execution.Offline or secured settings outweigh open online access; frequent command-line tools, environment variables, and MCP servers point to programming and automation workflows requiring real toolchains and controlled external systems.Source: §A.1 · p. 20, Figure 10 · p. 20

B Extended Results · B.1 SkillRouter Extended Routing Results

SkillRouter’s extended routing results show that performance declines unevenly as skill pools grow, while stronger models retain advantages on strict multi-skill coverage. GPT-5.5 leads SkillRouter on FC@10 at the full pool, and GPT-5.4 remains competitive there.

B.1 SkillRouter Extended Routing Results

  • GPT-5.5 exceeds SkillRouter by 16.0 points on FC@10 at the full pool of 79,141 skills.Table 4 reports extended routing results across skill pool sizes and distractor counts.Source: Table 4 · p. 21, §B.1 · p. 21
  • Table 4 covers skill routing across pool sizes and distractor counts, reporting scores as percentages and cost averaged per query.Source: Table 4 · p. 21
  • GPT-5.4 matches SkillRouter on Hit@1 at full scale while improving FC@10 by 8.0 points.Source: §B.1 · p. 21
  • All methods decline as candidate pools grow, with the decline varying across metrics and model sizes.Source: §B.1 · p. 21
  • FC@10 is more sensitive than Hit@1 because it requires every ground-truth skill to appear in the top 10 for majority multi-skill queries.Source: §B.1 · p. 21
  • GPT-5.4 mini performs well on small and medium pools but drops sharply at 50k and full scale, suggesting routing depends on model capability as libraries become highly confusable.Source: §B.1 · p. 21

C Experiment Details · C.1 SkillRouter

The experiments use shared agent settings across SkillsVote and baselines, while the SkillRouter study evaluates skill retrieval on expert-verified queries across candidate-pool scales using specified routing baselines, configurations, and metrics.

C Experiment Details

  • Across each benchmark, SkillsVote and baselines share the agent harness, solver, timeout, retry, and verifier settings, using three specified Codex CLI model-effort pairs.Built-in system skills and plugins are disabled for solver agents to reduce redundant context.Source: C Experiment Details · p. 22

C.1 SkillRouter

  • SkillRouter’s public evaluation set [78] contains 75 expert-verified queries: 24 single-skill and 51 multi-skill queries, scored against expert-verified target skills.The study uses the released Hard pool of 79,141 skills, including 780 LLM-generated distractors; auxiliary annotations are not used in reported metrics.Source: §C.1 · p. 22
  • Each evaluated pool retains all ground-truth skills and the listed distractors, then adds other released-pool skills to reach its target size and increase confusability.Table 5 reports the split sizes.Source: §C.1 · p. 22
  • The SkillRouter baseline encodes full skill text and retrieves 50 skills by embedding similarity, reranks the top 20, and scores the final top 10.It uses the released 0.6B embedding model and 0.6B reranker; API cost is reported for SkillsVote only.Source: §C.1 · p. 22
  • SkillsVote ranks exactly 10 existing skill files by relevance in a read-only Codex environment, without downstream task execution.Web search, built-in skills, plugins, writes, and command approvals are disabled; cost is averaged per completed query from input, cached-input, and output tokens using LiteLLM pricing.Source: §C.1 · p. 22
  • Routing is measured with Hit@1, Recall@10, and FC@10, with separate single-skill and multi-skill slices.Hit@1 counts any ground-truth skill at rank 1; Recall@10 is the fraction of ground-truth skills in the top 10; FC@10 requires all ground-truth skills in the top 10.Source: §C.1 · p. 22

C.2 Terminal-Bench 2.0

This section describes the Terminal-Bench 2.0 task set and avg@5 evaluation protocols for online evolution and two frozen-library transfer settings, alongside comparison baselines.

Dataset

  • Terminal-Bench 2.0 has 89 tasks—4 Easy, 55 Medium, and 30 Hard—across 16 categories, each with a unique container, reference solution, and executable tests.Categories include software engineering, debugging, security, machine learning, and scientific computing.Source: §C.2 · pp. 22–23

Evaluation metric

  • The benchmark metric is avg@5 Accuracy, averaging each task’s success rate over five runs before averaging across tasks.GPT-5.5 results exclude four cyber-risk-blocked tasks and are computed over the remaining 85 tasks.Source: §C.2 · p. 23

Offline skill evolution

  • Offline skill evolution collects experience from 48 Terminal-Bench Pro tasks, then transfers the frozen evolved library for recommendation across all 89 Terminal-Bench 2.0 tasks.The collection pass excludes two unreliable Docker images; ground truth assists attribution only and is neither exposed to the solver nor written directly into skills.Source: §C.2 · p. 23

Curated skill transfer

  • Curated transfer uses approximately 10k profiled open-source skills, restricted to validated, deduplicated, quality-analyzed skills from GitHub repositories with over 1k stars.Recommendation runs without evolution on all 89 tasks, five times independently, with avg@5 Accuracy reported.Source: §C.2 · p. 23

Online skill evolution

  • Online evolution starts with an empty library, enables recommendation and evolution, and evaluates five independent sequential task-order jobs with isolated libraries.Each job passes through all 89 tasks; results use avg@5 Accuracy.Source: §C.2 · p. 23

Baselines

  • Baselines are a solver without skills, reproduced ReasoningBank, and skill-creator prompted to evolve from completed trajectories instead of using SkillsVote attribution.All use five independent runs/jobs; ReasoningBank memory updates only at final outcomes and disables MaTTS. Environment failures may be retried up to three times, but verifier failures are final; timeout is four times the benchmark default.Source: §C.2 · pp. 23–24

C.3 SWE-Bench Pro … D.1.2 Profiling Skill Requirements, Quality, and Verifiability

The section defines the SWE-Bench Pro evaluation settings and describes SkillsVote’s million-scale skill corpus and operational profiling. It distinguishes offline recommendation from repository-isolated online evolution and specifies what the profiles assess.

C.3 SWE-Bench Pro

  • SWE-Bench Pro’s public split contains 731 human-verified and human-augmented tasks from 11 GPL repositories across Go, Python, JavaScript, and TypeScript [10].The repositories span business applications, B2B services, and developer tools; their task counts are 280 Go, 266 Python, 165 JavaScript, and 20 TypeScript.Source: §C.3 · p. 24
  • The evaluation reports avg@1 Resolve Rate: whether the single rollout for each task yields a patch that passes the verifier, averaged across tasks.Each task is run once, so avg@1 is the average solve rate.Source: §C.3 · p. 24
  • The offline curated-skill setting recommends from approximately 10k validated, deduplicated, quality-analyzed GitHub skills, filtered to repositories with over 1k stars, without evolution.It evaluates all 731 public tasks once and reports avg@1 Resolve Rate.Source: §C.3 · p. 24
  • Online skill evolution starts with an empty library, enables task-level recommendation and evolution, and runs 11 independent repository jobs with no skill-library transfer between repositories.All 731 tasks are run once, with overall avg@1 Resolve Rate reported.Source: §C.3 · p. 24
  • Baselines are w/o skills, ReasoningBank, and skill-creator, using the repository-level online protocol for the latter two and Terminal-Bench 2.0 timeout and retry settings.Memory updating and skill evolution settings also match those used on Terminal-Bench 2.0.Source: §C.3 · p. 24

D.1.1 Collecting a Million-Scale Agent Skill Corpus

  • SkillsVote collects a million-scale GitHub corpus as directory-level skill artifacts, retaining SKILL.md and optional scripts/, references/, and assets/ rather than relying on discovery metadata alone.SkillsMP and skills.sh offer discovery signals, but names, descriptions, and popularity do not establish runtime fit, resource completeness, coherent scope, or objective checkability.Source: §D.1.1 · p. 24

D.1.2 Profiling Skill Requirements, Quality, and Verifiability

  • SkillsVote profiles skills for runtime requirements, quality, and verifiability, with operational gates for recommendation and task synthesis.The profiles cover OS assumptions, write scope, privilege, externality, credentials, CLIs, MCP servers, environment variables; consistency, completeness, task orientation; and checkable success conditions, sandbox-controllable environments, and task construction cost.Source: §D.1.2 · p. 25

D.1.3 Synthesizing Verifiable Tasks from Agent Skills … E.2 Lifecycle of Harbor Evaluation Framework

SkillsVote profiles skill packages, synthesizes executable tasks from skills that pass verifiability, and connects recommendations, subtask attribution, and evidence-gated evolution. In Harbor, recommendation precedes task execution, while attribution and evolution follow the trial.

D.1.3 Synthesizing Verifiable Tasks from Agent Skills

  • Skills that pass verifiability are synthesized into Harbor tasks with clear instructions, reproducible environments, and executable verifiers, while unsuitable skills remain profiled corpus items.Agent–model runs record success rates, costs, traces, and verifier outcomes.Source: §D.1.3 · p. 25

D.2 Prompt Rendering Rules

  • Profiling, recommendation, attribution, and evolution use distinct prompts that constrain inputs and outputs and prevent recommendation from solving tasks or distilling benchmark-specific content.Attribution resumes the original solver session; evolution prompts support either editing an existing skill or creating a new one.Source: §D.2 · p. 25

D.3 Schema of Profiling Artifacts; D.4 Schema of Recommendation Artifacts

  • Profiling artifacts combine runtime requirements, quality, verifiability, and category rationale; recommendation artifacts specify exact skill names and concise guidance on coverage, combination, gaps, and boundaries.An empty recommendation is allowed only after effective search finds no relevant reusable skill.Source: §D.3 · pp. 25–26, §D.4 · p. 26

D.5 Schema of Attribution Artifacts

  • Attribution artifacts record trajectory subtasks, reusable exploration, judgment and attribution categories, and the linked skill and text spans actually relied on.The schema includes a factual subtask summary and evidence explanations for judgment and attribution.Source: §D.5 · p. 26

D.6 Schema of Evolution Artifacts

  • Evolution artifacts describe requests and actions, including the target skill, supporting subtasks, action rationale, edit summary, and any newly created skill directory.Create requests have no target skill; edit requests identify an existing skill.Source: §D.6 · p. 28

D.7 Aggregation of Subtasks

  • Evolution aggregates subtasks and admits only nonempty exploration with successful attribution; unlinked exploration creates a skill, while qualifying exploration linked to a skill is grouped into per-skill edit requests.Failed and uncertain subtasks do not trigger library updates.Source: §D.7 · pp. 29–30

E.1 Skill Corpus Validation and Profiling

  • Before recommendation or task synthesis, SkillsVote validates skill format, removes exact SKILL.md duplicates while retaining the earliest-commit copy, and profiles accepted packages for runtime, quality, and verifiability.The profiler can inspect the complete package, including scripts, references, assets, and auxiliary files.Source: §E.1 · p. 30

E.2 Lifecycle of Harbor Evaluation Framework

  • In Harbor, recommendation runs before the solver starts, and attribution and evolution run after the trial; SkillsVote skips intermediate failed attempts that Harbor will retry and leaves the solver workflow unchanged.The post-trial stages use trial results, agent artifacts, and verifier artifacts on the host side.Source: §E.2 · pp. 30–31

E.3 Dataset Preparation and Environment Setup … F Case Study

SkillsVote streamlines benchmark execution through prebuilt environments and shared orchestration, then operationalizes skill recommendation, task attribution, and evidence-gated evolution.

E.3 Dataset Preparation and Environment Setup

  • Prebuilt task images with Codex CLI 0.125.0, Node.js 22, and nvm 0.40.4 avoid repeated runtime installation and reduce concurrent downloads and trial preparation time.The images are built from Harbor’s original task images and used in task configuration.Source: §E.3 · p. 31

E.4 Experiment Configuration and Orchestration

  • A lightweight YAML launcher configures Harbor and SkillsVote modules, letting baselines, recommendation, and online or offline evolution share execution without Harbor code changes.Experiments are switched by changing YAML rather than copying execution scripts.Source: §E.4 · p. 31

E.5 Integration of Solver Agent

  • The solver retains Harbor’s Codex task execution and JSON logging, while setup skips CLI installation and disables system skills and plugins to isolate SkillsVote’s library.The prebuilt image already contains Codex 0.125.0; Harbor’s unified terminal tool remains enabled for session and status downloads.Source: §E.5 · p. 31

E.6 Integration of Skill Recommendation

  • Before solving, a separate agent searches candidate skills and produces concise usage guidance; validated nonempty recommendations install only selected skills, with failures falling back to all candidates.The recommendation agent searches skill documents and resources but does not solve the benchmark task; malformed structured output is retried up to three times.Source: §E.6 · pp. 31–32

E.7 Integration of Task Attribution

  • Attribution resumes the original Codex session to preserve harness context, structures trajectories into subtasks, and supplies controlled verifier evidence, with offline oracle prompts barring gold answers from reusable exploration.Online evidence may include test counts; offline oracle mode may additionally expose solution and verifier artifacts.Source: §E.7 · p. 32

F Case Study

  • A representative offline-transfer case shows a skill evolved from an Apache website task transferring persistent-service setup and end-to-end validation to an unseen Git-server deployment task.Source: Figure 12 · p. 33

E.8 Integration of Skill Evolution

  • Evolution consumes structured subtasks and admits only successful reusable exploration, routing unlinked discoveries to skill creation and existing-skill discoveries to per-skill edit requests.Each request has an independent working directory to isolate its read/write scope.Source: §E.8 · pp. 32–33

F.1 Offline Skill Transfer from TB-Pro to Terminal-Bench 2.0

These cases illustrate offline transfer of Apache task experience into reusable skills that are applied and updated across different Apache tasks, emphasizing operational patterns and runtime validation over task-specific answers. They also show practical execution constraints, including an `apt-get` lock conflict and incomplete Git-hook testing after Git installation fails.

F.1 Offline Skill Transfer from TB-Pro to Terminal-Bench 2.0

  • Apache-backed source trajectories yielded `ubuntu-apache-vhost`, encoding persistent configuration, service installation, and end-to-end runtime validation as reusable guidance.The skill also covers Ubuntu/Debian vhost setup, logging, access control, validation, and per-IP connection limits using `mod_qos` when needed.Source: §F.1 · pp. 33, 35
  • On a Terminal-Bench Pro logging and rate-limiting task, the agent validated Apache configuration and live HTTP behavior, with a denied path producing expected access and error logs.The skill captures persistent vhost setup and practical verification patterns, including testing a real denied directory for 403 behavior.Source: §F.1 · pp. 34–35
  • Offline evolution edited `ubuntu-apache-vhost` to add a `service`-command fallback when `systemctl` was unavailable and strengthen runtime validation.The update followed successful skill use during Apache service startup in a minimal environment.Source: §F.1 · p. 35
  • On an unseen Terminal-Bench 2.0 Git-webserver task, the skill-guided run used persistent Apache serving and end-to-end validation, unlike the baseline’s Node server without those safeguards.The run connected the Git post-receive hook to the served directory; a push followed by a curl request returned the expected file.Source: §F.1 · pp. 33, 37, 38, 39
  • An `apt-get` process holds `/var/lib/apt/lists/lock`, so the agent kills stuck processes and continues with generated setup files.The lock prevents `apt-get` from proceeding.Source: §F.1 · p. 40
  • The agent writes setup scripts and documentation but cannot fully exercise the Git hook because Git installation fails in the environment.Source: §F.1 · p. 40

F.2 Online Evolution on SWE-Bench Pro · G Prompts

This section specifies prompt workflows for skill profiling, recommendation and routing, post-execution attribution, and evidence-bounded evolution. These workflows use verified outcomes and reusable discoveries to guide skill edits or creation, while skipping unsupported changes.

F.2 Online Evolution on SWE-Bench Pro

  • The NodeBB repair separates route wiring, authenticated reproduction, Redis bootstrap, and API error serialization into attributed subtasks, editing skills only for reusable successful parts.The executable repro initially failed because Redis was unavailable, then confirmed the invite routes were absent before implementation.Source: §F.2 · pp. 40, 42
  • The agent implemented authenticated v3 invite issue, accept, and reject-or-rescind routes, updated the client and OpenAPI specification, and adjusted error handling to preserve canonical keys and return 403.The new endpoints use POST, PUT, and DELETE; the adjusted serializer behavior made the local repro pass.Source: §F.2 · p. 43
  • The local repro passed, but the verifier reported 421/427 private tests passing, leaving six failures.The remaining failures suggest unresolved edge cases or regressions in API status, translation, or permission behavior.Source: §F.2 · pp. 42–43

G.1 Skill Profiling

  • Profiling inspects each skill directory’s instructions and supporting files, reporting runtime requirements, quality signals, and suitability for automatically verifiable tasks.Runtime tags describe requirements while executing the skill, including OS, write scope, external dependencies, environment variables, executable commands, and MCP servers.Source: §G.1 · p. 44
  • Quality evaluation checks whether a skill has a consistent reusable purpose, complete required artifacts and dependencies, and actionable guidance for completing its task.A skill need not include scripts or dependencies when its workflow does not require them.Source: §G.1 · p. 45
  • Verifiability screening assesses whether success has a low-ambiguity programmatic check, its environment can be faithfully reproduced in Docker, and task instances and gold outcomes can be constructed at reasonable cost.Subjective judgments and approximate proxy metrics do not count as strong verification; external dependencies that cannot be faithfully sandboxed weigh against environment controllability.Source: §G.1 · pp. 45–46
  • Profiling is bounded to the target skill root: read SKILL.md first, inspect other files only as needed, and use null or [] when evidence is insufficient or mixed.Each reason must be a non-empty, evidence-based explanation of 3–5 sentences.Source: §G.1 · p. 46

G.2 Skill Recommendation

  • Recommendation searches candidate skills before execution and returns a useful, evidence-backed set with compact guidance for the downstream solver.Benchmark experiments use installable skill folders; recommendations over the curated library instead use markdown files under category directories, without treating referenced scripts or assets as candidate evidence.Source: §G.2 · pp. 47, 49, 50
  • The search decomposes the query into capability facets, expands keyword families, discovers candidates through filesystem search, and selectively reads and verifies relevant content.It iterates when core steps remain uncovered and compares similar skills for coverage, overlap, and intended use.Source: §G.2 · pp. 48–49
  • Recommendations favor fewer skills when coverage is clear, avoid name-only matches, and use optimized context to explain skill roles, combination, boundaries, and necessary coverage gaps.The context guides skill use rather than completing the user’s task or supplying its final deliverable.Source: §G.2 · p. 49
  • Recommendation returns exact existing skill names and concise downstream-use guidance, selecting an evidence-backed set that covers necessary steps without completing the user’s task.The optimized context explains each skill’s coverage, combination, and boundaries; an empty list is allowed only after meaningful search finds no reusable skill.Source: §G.2 · pp. 51–53
  • Recommendation decomposes the query into task facets, expands keyword families beyond literal terms, searches the filesystem, and inspects candidate skill markdown rather than referenced resources.Search should extend to intermediate steps such as setup, validation, and debugging, and continue when candidates do not cover the core requirements.Source: §G.2 · pp. 51–52

G.3 Skill Routing

  • Skill routing instead returns exactly top_k existing skill documents ranked by relevance with evidence-based reasons, rather than solver-facing usage guidance.The benchmark setting ranks relevant skill documents for a query and requires continued search until the best available top_k candidates are identified.Source: §G.3 · pp. 53–55
  • Skill routing retrieves exactly the requested number of skills, requiring capability evidence in their markdown and limiting results to real files under `skills_root`.The prompt says to return an empty list only when content search and candidate reading show no skill would meaningfully help.Source: §G.3 · pp. 55–56

G.4 Post-Execution Attribution

  • Post-execution attribution summarizes trajectories into subtasks and assigns outcome responsibility using execution traces, accessible skills, and the authoritative task-level verifier signal.The verifier evaluates the whole task, not any individual subtask in isolation.Source: §G.4 · p. 56
  • Each subtask is a minimal, semantically complete unit with one goal, one primary judge source, and at most one linked skill context, labeled by its final state.The prompt says not to split work merely because it involves many low-level commands.Source: §G.4 · p. 59
  • Under a trusted full-coverage verifier, task-level success is authoritative: intermediate subtasks may fail and be corrected, but all subtasks cannot succeed if the final task fails.Source: §G.4 · p. 60
  • Each subtask links at most one skill, viewed skills must appear across subtasks, and skill_refs include only knowledge spans actually used.If a skill was viewed but no specific span was used, skill_refs should be empty.Source: §G.4 · pp. 60–61
  • Attribution separates factual summaries from exploration, retaining reusable discoveries such as procedures, constraints, workarounds, recovery patterns, or decomposition, and otherwise setting exploration to null.Ordinary trial-and-error and one-off accidental discoveries that do not generalize are not reusable exploration.Source: §G.4 · p. 61

G.5 Skill Evolution

  • Evolution consumes successful attributed subtasks, aggregates reusable exploration, and uses skill references and the existing skill's scope as evidence for targeted changes.The existing skill is treated as mostly correct and coherent unless subtasks support a concrete modification.Source: §G.5 · pp. 62, 64
  • It edits within a coherent skill boundary, creates a separate skill for out-of-boundary knowledge, or skips weak evidence.Edit actions include correcting errors, adding reusable knowledge, or adding prerequisite guardrails.Source: §G.5 · pp. 62, 64
  • Keep changes local and directly supported by exploration, preserve correct guidance, and add only reusable procedural knowledge rather than task-specific facts or trajectory recaps.For new skills, the instructions call for a coherent, self-contained artifact; reusable discoveries should remain independently retrievable.Source: §G.5 · pp. 64, 67
  • Skip evolution when exploration is too task-specific, unstable, weakly supported, insufficiently reusable, or lacks evidence to choose safely between editing and creating.Prefer skipping over forcing unrelated or weakly supported knowledge into either action.Source: §G.5 · pp. 64–65
  • Skill edits distinguish error fixes for guidance that causes failure, knowledge additions for missing reusable steps or fallbacks, and prerequisite additions for missing conditions or guardrails.Prerequisite changes should make conditions explicit and operational while guarding the existing workflow.Source: §G.5 · pp. 65–66
  • Edit when a discovery concerns the same tool, workflow, problem, or operational scope; create a new skill when it crosses those boundaries or would make the existing skill too broad.Using two skills together in one task is not enough reason to merge them; when uncertain, prefer creation over forcing unrelated knowledge into an existing skill.Source: §G.5 · p. 66
  • The evolution output uses structured actions: create a skill with its directory path or skip, with the prompt requiring a null summary and a null path for skip.Created skill directories must be under the designated creation directory.Source: §G.5 · p. 69
  • The prompt reads subtasks together, extracts exploration-supported procedural knowledge, merges compatible findings, and checks the result for internal conflicts.This aggregation step aims to produce coherent reusable knowledge rather than preserve raw task traces.Source: §G.5 · p. 69
  • Create a skill only when exploration forms an independent reusable capability that should be retrieved on its own; skip weakly supported, unstable, narrow, or task-specific findings.The prompt also prefers skipping when exploration is not reusable enough to justify a new skill.Source: §G.5 · pp. 64, 69
  • When creating skills, synthesize focused, coherent, self-contained artifacts, separating semantically independent capabilities without fragmenting one workflow or combining unrelated domains.The prompt directs use of the skill-creator and standard skill-folder layout when creating or restructuring skills.Source: §G.5 · pp. 69–70
  • The skill-creator baseline resumes the completed trajectory in an editable skill workspace and preserves only reusable, transferable, actionable knowledge.Its prompt directs updating or creating skills when such knowledge exists; otherwise, it leaves the working directory unchanged.Source: §G.5 · p. 71

Related papers

Loading 2605.18401v2…