PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution

Hongyi Liu, Haoyan Yang, Tao Jiang, Bo Tang, Feiyu Xiong, Yuyu Luo, Zhiyu Li

arXiv:2605.18401v2cs.CLcs.AI

TL;DR

Raw trajectories and growing skill libraries are noisy, redundant, and difficult to govern for durable experience reuse. SkillsVote manages skills through task-conditioned recommendation, outcome attribution, and evidence-gated evolution, improving performance across Terminal-Bench 2.0 and SWE-Bench Pro, including a 2.7-point Resolve Rate gain.

  • Problem

    Raw trajectories are lengthy, noisy, environment-bound, and difficult to reuse, while growing skill ecosystems contain redundancy, uneven quality, and pollution risks.

  • Method

    SkillsVote governs Agent Skills through agentic library search before execution, outcome attribution afterward, and constrained updates admitting supported reusable experience.

  • Results

    Across all six benchmark-backbone pairs, SkillsVote improves overall scores and is best or tied for best online method, including a 2.7 pp avg@1 Resolve Rate gain on SWE-Bench Pro public.

  • Takeaways & Limitations

    SkillsVote supports conservative lifecycle management of persistent skills by linking task-conditioned use with attribution-controlled evolution across online and offline settings.

Abstract

from arXiv · show

Long-horizon LLM agents generate traces that could become reusable experience, but raw trajectories are noisy, local, and hard to govern. Agent Skills offer a structured artifact for combining procedural guidance, executable resources, and applicability boundaries. Yet open skill ecosystems contain redundant, uneven, environment-sensitive artifacts, and indiscriminate updates can pollute future context. We present SkillsVote, a lifecycle-governance framework for Agent Skills across collection, recommendation, attribution, and evolution. SkillsVote profiles a million-scale open source corpus for environment requirements, quality, and verifiability, and synthesizes tasks for verifiable skills. Before execution, it performs agentic library search over structured skill folders to expose instructional context. After execution, it decomposes trajectories into skill-linked subtasks, attributes outcomes to skill-guided execution, agent exploration, environment, and result signals, and admits only successful reusable discoveries to evidence-gated updates. Experiments on Terminal-Bench 2.0 and SWE-Bench Pro show that SkillsVote improves agent performance on challenging agentic coding benchmarks. The gains arise from two complementary pathways: online evolution over task streams at test time and offline transfer via frozen libraries built from either historical trajectories or curated open source skills.

1 Introduction

SkillsVote treats Agent Skills as governed lifecycle artifacts rather than raw trajectory reuse, coupling profiled-library recommendation, outcome attribution, and evidence-controlled evolution. It improves online evolution, offline transfer, and recommendation-controlled skill use on Terminal-Bench 2.0 and SWE-Bench Pro.

  • Motivation: Raw trajectories are lengthy, noisy, environment-bound, and difficult to reuse because they conflate robust strategies with incidental state.Agent Skills package procedural instructions, executable resources, dependency boundaries, and applicability conditions into more auditable, compact artifacts.
  • Motivation: Open skill ecosystems exhibit scale, redundancy, uneven quality, and safety risks, while weakly related or poorly written skills can degrade agent performance.Skill benefits depend on task, domain, and retrieval setting, making indiscriminate library growth hazardous.
  • Framework: Before execution, agentic search selects a small, relevant, low-redundancy skill set; afterward, attribution relates outcomes to skills, exploration, environment, and evaluation signals.The framework uses a profiled structured library and decomposes trajectories into skill-linked subtasks for reusable updates.
  • Framework: SkillsVote connects collection and governance, recommendation, outcome attribution, and controlled evolution in an Agent Skill lifecycle framework.Its loop uses attribution to constrain library updates and reduce indiscriminate evolution.
  • Evaluation: SkillsVote profiles more than one million open source Agent Skills and improves online evolution, offline transfer, and recommendation-controlled skill use on Terminal-Bench 2.0 and SWE-Bench Pro.Experiments examine recommendation against direct library exposure, offline transfer from historical trajectories, and online accumulation during test-time task streams.

2 Related Work

Related work progresses from unstructured experience records toward executable artifacts and skill-centered agent evolution. As skills become installable ecosystem objects, research increasingly addresses their organization, retrieval, evaluation, and governance.

  • Evolution of Agent Experience Learning: Experience learning has advanced from unstructured cases and examples to semi-structured workflows, SOPs, principles, heuristics, and executable artifacts.These approaches include few-shot trajectories, exemplars, human-curated interaction records, workflows, and strategy-level abstractions.
  • Agent Skill Ecosystems, Retrieval and Evaluation: Installable, shareable Agent Skills shift the research problem from skill authoring toward governing and using open ecosystems.AgentSkillOS and SkillNet organize skills as ecosystem objects, while SkillsBench, SkillCraft, SkVM, and SkCC study related ecosystem capabilities.
  • Agent Skill Ecosystems, Retrieval and Evaluation: Skill retrieval research includes routing over full skill bodies and direct corpus interaction over source documents rather than relying only on names, descriptions, or embeddings.SkillsVote applies filesystem-native inspection to governed skill folders and produces compact guidance for combining selected skills.
  • Skill-Centric Agent Self Evolution: Skill-centric self-evolution studies retrieval decisions, skill use, behavioral distillation, library revision, and reusable artifacts derived from session or trajectory evidence with verifier or environment feedback.Some approaches train policies for these decisions, while others keep the base model fixed and convert coarse evidence into skill artifacts.

3 Approach

SkillsVote governs Agent Skills as lifecycle artifacts by controlling both which skills enter solver context and which execution evidence can update the library. It combines agentic recommendation, subtask-level attribution, and conservative evidence-gated evolution to expose relevant guidance and admit only successful, reusable discoveries.

  • Recommendation: Before solving, SkillsVote searches the local library, selectively reads candidate skills and resources, and exposes a compact complementary set with a usage guide.Selection considers task coverage, target-environment fit, and complementary guidance rather than relying on the full library, metadata-only routing, or a single-step decision.
  • Attribution: SkillsVote segments trajectories into subtasks with one standalone objective, one primary evaluation signal, and at most one associated skill context.Splitting occurs when the objective, evaluation signal, or skill context changes, making units local enough for responsibility assignment and abstract enough to capture reusable procedures, constraints, and recovery patterns.
  • Attribution: Each subtask records outcome evidence, assigns responsibility across skill-guided execution and exploration, and extracts only reusable knowledge while retaining failures diagnostically.Reusable deltas include missing procedures, preconditions, and recovery patterns; ordinary trial-and-error, task-specific constants, and repetitive operational details are discarded.
  • Evolution: Only successful units containing reusable exploration are admissible for evolution; related evidence is aggregated before editing to strengthen one update instead of producing duplicates.Failed, uncertain, or weakly supported evidence may support diagnosis but cannot directly authorize a skill update.
  • Evolution: Aggregated evidence is routed to the smallest justified skill edit, a new skill, or no change when evidence is weak, redundant, or misaligned.Evolution remains localized to the relevant skill boundary and expresses reusable procedural knowledge rather than a trajectory recap.

4 Experiments

The evaluation is organized around three lifecycle questions, including whether frozen skill libraries improve performance on unseen tasks. One question specifically examines offline transfer from historical trajectories or curated open source skills.

  • The experiments are organized around three lifecycle questions.
  • Offline skill transfer: Offline skill transfer asks whether frozen libraries improve performance on unseen tasks.The libraries are built from either historical trajectories or curated open source skills.
  • Offline skill transfer: The offline-transfer comparison considers frozen libraries derived from historical trajectories and curated open source skills.

2. Online skill evolution. Can SkillsVote accumulate useful skills over a sequential task stream?

SkillsVote accumulates useful skills through online evolution, improving performance across all tested benchmark–backbone pairs while matching or surpassing online baselines. Frozen trajectory-derived libraries also transfer strongly, and recommendation reduces harmful skill exposure by filtering noisy or weakly related skills.

  • Main results: Across all six benchmark–backbone pairs, SkillsVote improves over no skills and is the best online method or tied for best in every pair.On Terminal-Bench 2.0, avg@5 Accuracy rises by ↑2.6 pp, ↑1.1 pp, and ↑0.9 pp for GPT-5.2, GPT-5.4 mini, and GPT-5.5; on SWE-Bench Pro public, avg@1 Resolve Rate rises by ↑2.7 pp, ↑2.6 pp, and ↑1.2 pp.
  • Offline transfer: Historical TB-Pro skills provide the strongest offline Terminal-Bench 2.0 transfer, improving GPT-5.2, GPT-5.4 mini, and GPT-5.5 by ↑7.8 pp, ↑5.8 pp, and ↑1.4 pp.These trajectory-derived skills capture reusable terminal procedures beyond the historical tasks used to build the library.
  • Offline transfer: Offline reuse is inconsistent across frozen libraries, benchmarks, and backbones: Curated improves Terminal-Bench 2.0 for GPT-5.2 and GPT-5.4 mini by ↑3.7 pp and ↑4.0 pp, but declines by ↓1.7 pp in another setting and helps only GPT-5.4 mini on SWE-Bench Pro.TB-Pro is used only for Terminal-Bench 2.0 transfer, while Curated is a frozen library of 10K curated open source skills.
  • Skill routing: As the candidate pool grows from 1k to 79,141 skills, FC@10 declines for all methods, but GPT-5.5 remains above SkillRouter and reaches 62.7 at full scale versus 46.7.The larger pools increase difficulty because complete coverage becomes harder in more confusable libraries.
  • Recommendation: Recommendation converts direct exposure’s +3.3/−6.7 task-level gain/loss profile into a balanced +6.0/−6.0 profile by filtering sparse, underspecified, or weakly related skills.This noise-filtering effect is most evident in the early online regime.
  • Offline evolution: Offline evolution separates benchmark performance from library utility: Terminal-Bench Pro performance is nonmonotonic, while frozen libraries transfer increasingly well to unseen Terminal-Bench 2.0 Hard tasks.Post-task attribution uses ground-truth and verifier signals only after completion to identify successful, reusable, attributable trajectory components.

5 Conclusion · Appendix · A Extended Analysis

The conclusion frames SkillsVote as lifecycle governance for Agent Skills, connecting corpus profiling, task-conditioned recommendation, outcome attribution, and evidence-gated evolution. The appendix organizes extended analyses, results, experiment details, approach and implementation schemas, case studies, and prompts supporting these components.

  • 5 Conclusion: SkillsVote manages Agent Skills across collection, recommendation, attribution, and evolution to reduce irrelevant retrieval and weakly supported library updates.It links a million-scale open-source corpus with execution-readiness profiling, task-conditioned recommendation, subtask-level attribution, and evidence-gated evolution.
  • Appendix: The extended-analysis appendix includes skill characteristics, extended results, SkillRouter routing results, and experiment details for Terminal-Bench 2.0 and SWE-Bench Pro.The listed sections cover SkillRouter, benchmark-specific experiment details, and extended result reporting.
  • A Extended Analysis: The appendix separately lists extended analysis, experiment details, approach details, implementation details, case studies, and prompts as supporting material for the framework.Its contents include dedicated sections for SkillRouter, benchmark setup, artifact schemas, lifecycle integration, transfer cases, and operational prompts.
  • Appendix: The approach-details appendix specifies the open-source skill corpus, profiling, prompt rendering, and schemas for profiling, recommendation, attribution, and evolution artifacts.It also lists subtasks aggregation as a dedicated approach component.
  • Appendix: The implementation-details appendix covers corpus validation and profiling, Harbor evaluation, dataset and environment setup, experiment orchestration, solver integration, recommendation, attribution, and evolution.These entries describe how the lifecycle components are integrated into the evaluation framework.
  • Appendix: The case-study appendix reports offline skill transfer from TB-Pro to Terminal-Bench 2.0 and online evolution on SWE-Bench Pro.These cases correspond to the paper’s two stated transfer pathways: offline transfer and online evolution.
  • Appendix: The prompts appendix provides prompts for skill profiling, recommendation, routing, post-execution attribution, and skill evolution.The prompt sections span the lifecycle from profiling and pre-execution selection through post-execution analysis and updates.

A.1 Skill Characteristics

SkillsVote’s profiling shows an open-skill corpus dominated by coherent guidance units but limited in completeness, verifiability, and constructability. The ecosystem is concentrated in development and meta-skills, while runtime requirements favor Linux/macOS, writable scopes, and offline or secured settings.

  • Quality and verifiability: Fewer skills satisfy completeness, success verifiability, and task constructability than consistency and orientation.The profiling results indicate that many skills provide coherent reusable guidance, but fewer directly support low-ambiguity benchmark construction.
  • Categories: More than 290K profiled skills are dominated by Development, followed by AgentMeta and Tools.Development and AgentMeta also show the strongest co-occurrence pattern, linking development-oriented skills with higher-level agent guidance.
  • Filtration: More than 1.68M collected skills undergo sequential format validation, deduplication, and quality analysis, with stricter filtering shifting only a limited share toward higher-star repositories.Repository stars remain concentrated in the long tail below 100 stars throughout curation.
  • Skill names: skill-creator appears more than 5K times and frontend-design more than 3K times among the 20 most repeated skill names.The repeated-name distribution is highly concentrated, with meta-skills dominating alongside broad workflows such as debugging, testing, and document processing.
  • Runtime requirements: Approximately one million evaluated skills predominantly target Linux and macOS, writable execution scopes, and offline or secured access.Frequent command-line tools, environment variables, and MCP servers also characterize the runtime-requirement profile.

B Extended Results · B.1 SkillRouter Extended Routing Results

The extended SkillRouter results evaluate routing across skill-pool sizes and distractor counts, showing that larger candidate pools reduce performance unevenly across metrics and model sizes. GPT-5.5 has the strongest FC@10 across pool sizes, while FC@10 is stricter and more sensitive than Hit@1 for most multi-skill queries.

  • B.1 SkillRouter Extended Routing Results: The routing study defines skill-pool sizes as a central evaluation dimension for SkillRouter.Appendix C.1 specifies the split construction and evaluation protocol.
  • B.1 SkillRouter Extended Routing Results: A word cloud summarizes the 20 most frequently repeated skill names in the open-skill ecosystem.This visualization provides ecosystem-level context for the routing study.
  • B.1 SkillRouter Extended Routing Results: Table 4 reports SkillRouter routing results across skill-pool sizes and distractor counts.Scores are percentages, while cost is averaged per query.
  • B.1 SkillRouter Extended Routing Results: All methods decline as the candidate pool grows, with uneven declines across metrics and model sizes.The extended results cover the full range of skill-pool sizes.
  • B.1 SkillRouter Extended Routing Results: FC@10 is the most sensitive metric because it requires every ground-truth skill to appear in the top 10.This makes FC@10 stricter than Hit@1 for the majority of multi-skill queries.
  • B.1 SkillRouter Extended Routing Results: 79,141 skills mark the full-pool setting where GPT-5.5 remains 16.0 points above SkillRouter on FC@10.GPT-5.5 maintains the strongest FC@10 across all pool sizes.

C Experiment Details · C.1 SkillRouter

The experiments standardize the agent harness and solver settings, then evaluate SkillRouter routing on a filtered 75-query benchmark across increasingly large and confusable skill pools. SkillsVote is compared with a fixed embedding-and-reranking baseline using defined top-10 retrieval metrics.

  • C Experiment Details: Across each benchmark, SkillsVote and baselines share the agent harness, solver configuration, timeout, retry, and verifier settings.Runs use Codex CLI 0.125.0 with GPT-5.2, GPT-5.4 mini, and GPT-5.5 under specified reasoning efforts; built-in skills and plugins are disabled.
  • C.1 SkillRouter: The SkillRouter evaluation set contains 75 expert-verified queries: 24 single-skill and 51 multi-skill queries.The released Hard pool contains 79,141 skills, including 780 LLM-generated distractors; expert-verified skill sets define scoring targets.
  • C.1 SkillRouter: Each evaluated pool retains all ground-truth skills while adding distractors and other released skills until reaching its target size.This construction preserves scorability for every query while increasing candidate-library size and confusability.
  • C.1 SkillRouter: The SkillRouter baseline retrieves 50 skills by embedding similarity, reranks 20, and scores the final top 10 using released 0.6B models.Skill documents are encoded from full text formatted as name, description, and body; API cost is reported only for SkillsVote.
  • C.1 SkillRouter: SkillsVote ranks exactly 10 existing markdown skill files in a read-only Codex environment without execution, web search, plugins, writes, or command approvals.The evaluated models are GPT-5.5, GPT-5.4, and GPT-5.4 mini, all with xhigh reasoning effort.
  • C.1 SkillRouter: Routing is evaluated with Hit@1, Recall@10, and FC@10, including separate single-skill and multi-skill slices.Hit@1 detects any ground-truth skill at rank 1; Recall@10 measures the ground-truth fraction in the top 10; FC@10 requires all ground-truth skills there.

C.2 Terminal-Bench 2.0

Terminal-Bench 2.0 evaluates agents on 89 verified terminal tasks spanning real technical workflows and 16 categories, using leaderboard avg@5 Accuracy. SkillsVote is tested through online evolution, offline evolution, and curated skill transfer, with controlled retries, baselines, and task exclusions.

  • Benchmark: Terminal-Bench 2.0 contains 89 tasks—4 Easy, 55 Medium, and 30 Hard—across 16 categories inspired by real terminal-centric workflows.Each task has a unique container environment, human-written reference solution, and executable verification tests.
  • Configuration and baselines: The setup permits up to 3 retries for environment failures but treats post-execution verifier failures as final outcomes, while comparing against w/o skills, ReasoningBank, and skill-creator.Four cyber-risk-blocked tasks are excluded from every GPT-5.5 setting, leaving 85 tasks for those averages.
  • Evaluation: The benchmark reports avg@5 Accuracy, averaging each task’s success rate over five independent runs before averaging across tasks.Accuracy records whether a task is solved.
  • Offline skill evolution and transfer: Offline studies include one recommendation-and-evolution pass over 48 reliable Terminal-Bench Pro tasks and recommendation-only transfer from approximately 10k curated GitHub skills.The curated library applies format validation, deduplication, quality analysis, and a repository threshold exceeding 1k stars.
  • Online skill evolution: SkillsVote evaluates online evolution from an empty library over the default 89-task order, using five independent jobs with isolated libraries.Each job performs one sequential pass with task-level recommendation and evolution enabled.

C.3 SWE-Bench Pro … D.1.2 Profiling Skill Requirements, Quality, and Verifiability

SkillsVote evaluates skill transfer and online evolution on a verified 731-task SWE-Bench Pro split, while profiling open skill ecosystems for execution requirements, quality, and verifiability. Its corpus pipeline converts marketplace-scale discovery artifacts into execution-ready skills with explicit environmental assumptions and checkable success conditions.

  • C.3 SWE-Bench Pro: SWE-Bench Pro contains 731 verified, human-augmented long-horizon software-engineering tasks from 11 public GPL repositories spanning four language groups.The repositories cover business applications, B2B services, and developer tools.
  • C.3 SWE-Bench Pro: Evaluation reports avg@1 Resolve Rate, the average single-rollout solve rate for patches that pass each task’s verifier.Each task is run once, so avg@1 is the average solve rate across tasks.
  • D Approach Details: Offline curated transfer uses approximately 10k profiled skills after format validation, deduplication, quality analysis, and filtering to GitHub repositories with more than 1k stars.This setting performs recommendation only, without skill evolution, across all 731 public tasks.
  • C.3 SWE-Bench Pro: Online skill evolution begins with an empty library, enables task-level recommendation and evolution, and runs 11 repository-specific jobs to reuse experience within each codebase.All repositories are executed once, with overall avg@1 Resolve Rate reported across the 731 tasks.
  • D.1.1 Collecting a Million-Scale Agent Skill Corpus: Open skill marketplaces provide search, categorization, popularity, and installation signals, but these metadata do not establish runtime fit, resource completeness, coherent scope, or objective checkability.Skill utility also depends on the task, domain, and corpus quality.
  • D.1.2 Profiling Skill Requirements, Quality, and Verifiability: SkillsVote profiles skills along runtime, quality, and verifiability dimensions to produce execution-ready artifacts.Runtime profiling captures operating-system assumptions, write scope, privilege, externality, credentials, CLIs, MCP servers, and environment variables; quality checks consistency, completeness, and task orientation; verifiability checks low-ambiguity success conditions, sandbox controllability, and task construction cost.

D.1.3 Synthesizing Verifiable Tasks from Agent Skills … E.2 Lifecycle of Harbor Evaluation Framework

SkillsVote governs Agent Skills through verifiable task synthesis, structured profiling, recommendation, attribution, and evidence-gated evolution. Its Harbor integration places recommendation before execution and attribution/evolution after verifier-backed trial completion while preserving Harbor’s solver workflow.

  • D.1.3 Synthesizing Verifiable Tasks from Agent Skills: For verifiable skills, SkillsVote synthesizes Harbor tasks with clear instructions, reproducible environments, executable verifiers, and recorded success rates, costs, traces, and outcomes.Preference-driven, open-world, and hardware-intensive skills remain profiled corpus items rather than executable task instances.
  • D.2 Prompt Rendering Rules: Prompt rendering separates profiling, recommendation, attribution, and evolution roles, constraining recommendation to capability matching and evolution to creating or editing designated skill directories.Attribution resumes the original solver session and receives skills, working paths, and verifier signals; offline mode additionally exposes solution and verifier artifacts.
  • D.3 Schema of Profiling Artifacts: Profiling artifacts jointly encode runtime requirements, quality, verifiability, categories, and rationales through a flattened SkillEvaluationRubric.Runtime fields include operating systems, write scope, privileges, external dependencies, environment variables, binaries, MCP servers, and evidence reasons.
  • D.4 Schema of Recommendation Artifacts: Recommendation artifacts require exact, duplicate-free skill directory names and concise optimized_context describing coverage, composition, gaps, and usage boundaries without solving the task.An empty recommendation is allowed only after effective search finds no relevant reusable skill.
  • D.5 Schema of Attribution Artifacts: Attribution artifacts decompose trajectories into subtasks and record reusable exploration, judge signals, outcome attribution, linked skills, and the referenced skill text spans actually used.Each subtask includes an independent goal and factual summary, while skill references identify files, line spans, capabilities, and actual use.
  • D.6 Schema of Evolution Artifacts: Evolution artifacts represent create, edit, or skip actions with supporting subtasks, rationales, target skills, summaries, and resulting skill paths.Successful attribution categories can trigger updates, whereas failed and uncertain categories are skipped.
  • D.7 Aggregation of Subtasks: Before evolution, SkillsVote aggregates subtasks and admits only nonempty exploration from successful categories, creating new skills without editable links and grouping linked explorations into one edit request per old skill.Failed and uncertain subtasks do not trigger library updates.
  • E Implementation Details; E.1 Skill Corpus Validation and Profiling; E.2 Lifecycle of Harbor Evaluation Framework: SkillsVote validates, deduplicates, and profiles skill packages before recommendation or task synthesis, then integrates recommendation, attribution, and evolution into Harbor’s trial lifecycle.Recommendation runs before solver startup; attribution and evolution run host-side after trial completion using agent and verifier artifacts, while Harbor’s solving workflow remains unchanged.

E.3 Dataset Preparation and Environment Setup … F Case Study

SkillsVote streamlines benchmark execution with prebuilt environments, shared YAML orchestration, controlled solver integration, and staged skill recommendation. It then attributes trajectories through resumed native sessions and evolves the library only from validated, successful exploration, with an offline-transfer case illustrating persistent-service setup and validation reuse.

  • E.3 Dataset Preparation and Environment Setup: Prebuilt images install Codex CLI 0.125.0, nvm 0.40.4, and Node.js 22, reducing repeated downloads and shortening concurrent trial preparation.The images are built from original Harbor task images and skip runtime CLI installation.
  • E.4 Experiment Configuration and Orchestration: A lightweight YAML launcher unifies Harbor configuration with SkillsVote modules, allowing baselines, recommendation, and evolution experiments to change settings without modifying Harbor code.The same launcher supports online and offline evolution while avoiding duplicated execution scripts.
  • E.5 Integration of Solver Agent: The solver preserves Harbor’s Codex execution logic while using preinstalled Codex, JSON logging, unified terminal execution, and disabled system skills and plugins for controlled measurement.Preparation creates agent directories instead of reinstalling the CLI, and configuration disables extra prompt injection from built-in skills and plugins.
  • E.6 Integration of Skill Recommendation: Recommendation searches a read-only candidate library, selects relevant and nonredundant skills, validates structured output with up to three retries, and passes selected guidance to the solver.On repeated failure or installation failure, the system records the error and falls back to copying all candidate skills.
  • E.7 Integration of Task Attribution: Attribution resumes the original Codex session inside the native harness, preserving contextual details before converting the trajectory and verifier result into structured subtasks.Online mode exposes task-level test counts, while offline oracle mode can additionally expose solution and verifier artifacts under restrictions against reusable gold answers.
  • E.8 Integration of Skill Evolution: Evolution consumes only structured subtasks and filters out failed, uncertain, or nonreusable exploration, so only successful exploration can trigger skill-library updates.Successful exploration creates new skills when unlinked or forms independent edit requests when linked to existing skills.
  • F Case Study: An offline-transfer case shows a skill evolved from an Apache website task transferring persistent-service setup and end-to-end validation to an unseen Git-server deployment task.This case illustrates reuse of evolved procedural guidance across tasks.
  • E.8 Integration of Skill Evolution: Each evolution request isolates its working scope, validates structured JSON output, records logs, archives the session, and backs up existing skills before edits.Create and edit requests use separate local directories, with edit requests copying the target skill into an editable location.

F.1 Offline Skill Transfer from TB-Pro to Terminal-Bench 2.0

Offline evolution distilled successful Apache configuration and validation trajectories from Terminal-Bench Pro into the reusable ubuntu-apache-vhost skill. On an unseen Terminal-Bench 2.0 task, recommendation-enabled execution reused this operational pattern and extended it with a service-start fallback and stronger runtime validation.

  • Skill creation: Successful Terminal-Bench Pro exploration was distilled into ubuntu-apache-vhost guidance for persistent Apache configuration, service startup, and runtime validation.The skill captures persistent ports, site-specific settings, explicit document roots and logs, syntax validation, and custom 403 behavior.
  • Transfer to Terminal-Bench 2.0: On the unseen configure-git-webserver task, the recommended skill transferred Apache serving, Git-hook deployment, stable service setup, and final URL validation without copying the source solution.The evolved run served the endpoint through Apache and connected the Git post-receive hook to the served directory, whereas the baseline used a lightweight Node server without persistent service setup or final runtime validation.
  • Skill evolution: Offline evolution updated ubuntu-apache-vhost after systemctl was unavailable, adding a service-command fallback while strengthening intermediate and final runtime validation.The agent followed the skill to launch Apache, discovered the unavailable command, and extracted the fallback exploration into the skill.
  • Transfer to Terminal-Bench 2.0: The transferred execution established a *:8080 Apache vhost and verified deployment by pushing a local commit and retrieving hello world from the served URL.apache2ctl -S reported the vhost, and curl http://127.0.0.1:8080/hello.html returned the expected content.

F.2 Online Evolution on SWE-Bench Pro · G Prompts

The online SWE-Bench Pro evolution run decomposes a NodeBB repair into skill-linked subtasks, validates reusable changes through authenticated repros, and gates updates on outcome evidence. The prompts profile skills for environment, quality, and verifiability, while post-execution attribution distinguishes whole-task verification from reusable exploration.

  • F.2 Online Evolution on SWE-Bench Pro: The NodeBB repair separates route wiring, authenticated reproduction, Redis bootstrap, and API error serialization into attributed subtasks, admitting only successful reusable parts for skill editing.The recommended skills cover distinct repair components rather than treating the trajectory as one undifferentiated outcome.
  • F.2 Online Evolution on SWE-Bench Pro: After Redis startup and response-status debugging, the authenticated local repro passed, but private verification remained incomplete at 421/427 tests.The remaining six failures were attributed to possible edge cases, global API status or translation behavior, or permission semantics.
  • F.2 Online Evolution on SWE-Bench Pro: The agent implemented authenticated v3 POST, PUT, and DELETE group-invitation endpoints with permission checks, event logging, controllers, and routes.The implementation added issueInvite, acceptInvite, and rejectInvite in src/api/groups.js and exposed them at /api/v3/groups/:slug/invites/:uid.
  • G.1 Skill Profiling: The profiling prompt evaluates each skill directory by inspecting SKILL.md and supporting files for runtime requirements, quality signals, and suitability for automatically verifiable tasks.A skill root includes SKILL.md plus optional scripts, references, and assets.
  • G.1 Skill Profiling: Environment profiling records operating systems, write scope, privilege, externality, environment variables, executable bins, MCPs, and evidence-based justification.Write scope distinguishes read-only, workspace-limited, and system-level effects; externality distinguishes offline, online, and secured workflows.
  • G.1 Skill Profiling: Quality evaluation checks whether a skill has one consistent purpose, complete referenced artifacts and dependencies, and actionable task-oriented guidance.Consistency, completeness, and orientation are each evaluated as boolean rubric dimensions.
  • G.1 Skill Profiling: Verifiability screening assesses whether success is programmatically checkable, the environment is reproducibly controllable in Docker, and tasks with verifiers can scale affordably.Accepted checks include tests, schema validation, file diffs, compiler or runtime checks, and API, database, or DOM assertions.
  • G.4 Post-Execution Attribution: Post-execution attribution treats the verifier as a task-level success signal while defining exploration as a reusable delta such as a procedure, constraint, workaround, or recovery pattern.A trusted verifier evaluates the complete task goal, so intermediate subtask failures do not necessarily preclude overall task success.
Loading 2605.18401v2…