PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History

Zhiwei Li, Yong Hu

arXiv:2606.08671v3cs.LG

TL;DR

Continual skill maintenance needs persistent decision history because final artifacts omit the rationale and evidence behind prior revisions. SkillHone records that history and improves seeded skills by an average of 18.8 points across seven internal scenarios while exceeding deep-research and skill-development baselines.

  • Problem

    Existing skill-improvement methods retain final artifacts but not the decision history needed for continual maintenance across development sessions.

  • Method

    SkillHone is a general harness that uses role-separated subagents and redacted practice feedback to record diagnoses, revisions, evidence, and outcomes.

  • Results

    SkillHone exceeds iterative, reflective, and deep-research baselines and improves seeded skills by an average of 18.8 points across seven recurring internal scenarios.

  • Takeaways & Limitations

    Persistent decision history lets later agents improve skills without re-deriving prior decisions, and the skill bundle transfers across execution backbones without further optimization.

  • Takeaways & Limitations

    SkillHone evolves a single skill in isolation and does not address jointly evolving multiple interdependent skills.

Abstract

from arXiv · show

Agent skills extend language-model agents with task-specific procedures, scripts, and references, but the tasks and environments they target continually change. Existing methods improve skills in bounded runs and retain only the final artifact, discarding the decision history that later agents need to interpret prior revisions, evaluations, and rejected alternatives. We introduce SkillHone, a harness for continual agent skill evolution grounded in persistent decision history. SkillHone pairs skill revisions with evaluation-side evidence that supplies practice feedback, recording structured histories of diagnoses, revisions, evidence, and outcomes. Role-separated subagents run candidate skills on practice probes with redacted reporting and propose revisions informed by prior decisions, enabling cross-session refinement without rediscovering past rationale. On deep-research benchmarks, SkillHone runs without a pre-integrated search stack and outperforms the commercially backed deep-research agent by 15.8 points on GAIA and 3.2 points on WebWalkerQA-EN, while also exceeding prior skill-evolution methods. We further deploy SkillHone on internal tool-mediated analysis scenarios, where it improves accuracy by an average of 18.8 points across seven settings.

1 Introduction

SkillHone addresses the gap between artifact-centered skill improvement and continual maintenance by preserving decision history alongside evaluation evidence. Its role-separated harness enables later agents to refine skills across sessions, and the paper reports improvements on benchmarks and deployments.

  • SkillHone: SkillHone records diagnoses, proposed revisions, evaluation evidence, and outcomes for both evolving skills and the assets used to evaluate them.The harness treats revisions and evaluation evidence as decision-bearing rather than retaining only the latest artifact.
  • Method: Role-bounded subagents generate linked decision records under separated permissions, while optimization subagents use prior records without accessing hidden probe targets or validators.This design supports continued refinement without re-deriving prior decisions.
  • Motivation: Long-lived skills need persistent decision history so later agents can understand revisions, rejected alternatives, diagnoses, evaluation evidence, and outcomes.Artifact-only evolution can cause context loss and repeated prior optimizations.
  • Evaluation: SkillHone improves skill development across public benchmarks and internal tool-mediated deployments.The evaluation includes raw open-web deep-research benchmarks and deployment scenarios.

2 Method

SkillHone is a harness for improving a fixed skill artifact across repeated sessions by separating optimization from evaluation and preserving auditable decision history. Its repositories, permission boundaries, and persistent records let later agents use redacted feedback to refine skills without repeating prior reasoning.

  • Skill and repositories: SkillHone maintains a loadable skill artifact while keeping the model backend and external tools fixed across repeated development sessions.The skill packages instructions, procedures, references, and output conventions for a task class.
  • Skill and repositories: Two linked repositories separate the revisable skill bundle from evaluation assets that provide practice feedback for improvement.The evaluation repository includes probes, oracle targets, validators, traces, redacted reports, and related assets.
  • Dispatch and permission boundaries: Role-bounded optimization and evaluation dispatches exchange redacted evidence while recording diagnoses, revisions, and outcomes as auditable decision history.Optimization can change the skill but cannot access unredacted evaluation data; evaluation can inspect targets and traces but cannot write to the skill repository.
  • Persistent decision history: Each decision record links a diagnosis, candidate revision, redacted evaluation evidence, and outcome, rather than recording only a version diff.Outcomes can accept, reject, request further changes, or defer a diagnosis.
  • Persistent decision history: Later optimization subagents retrieve relevant prior records to revisit diagnoses, revise rejected patches under new evidence, or avoid redundant edits.This persistent memory supports continued improvement and auditing without re-deriving the same diagnoses.

3 Experiments

SkillHone is evaluated on public deep-research benchmarks through curated-search and raw-open-web comparisons, including transfer and ablation studies. It achieves the strongest reported benchmark results and improves recurring internal tool-mediated analysis skills.

  • Public benchmark results: +20.5/+28.3 points over Skill-Creator and +14.2/+13.4 over Hermes-SE distinguish SkillHone on GAIA/WebWalkerQA-EN.All raw open-web systems share the same starting skill pool and portable skill-bundle interface, differing in bundle development.
  • Transfer evaluation: 72.4% on GAIA is achieved after direct transfer to Claude Sonnet 4.6 without additional optimization, exceeding Hermes-SE/Existing-Skills/Skill-Creator by 10.2/15.7/24.4 points.The transfer study runs each skill bundle directly on the transfer backbone.
  • Ablation study: 13.4/10.9 points are lost on GAIA/WebWalkerQA-EN without decision history, compared with 6.4/5.3 points without role separation.Both mechanisms are ablated under the raw open-web setting with the same Qwen3.6-35B-A3B execution backbone.
  • Internal deployment study: 18.8 points is SkillHone’s average accuracy improvement across seven internal tool-mediated analysis scenarios.Six of seven skills improve, while list filtering remains stable at the aggregate level; the largest gains occur where procedures underspecify reusable analysis steps.

4 Related Work

Prior work studies reusable agent skills, skill acquisition, language-system optimization, and multi-agent workflows. SkillHone differs by preserving optimization context and using role-separated agents for revision and evaluation under bounded permissions.

  • Agent skills and skill acquisition: Agent skills package task-specific procedures, scripts, references, and conventions into reusable artifacts that language-model agents can load on demand.
  • Agent skills and skill acquisition: Existing benchmarks and acquisition methods evaluate or produce skills through general tasks, software engineering, exploration, feedback, reinforcement learning, and domain-specific construction.
  • Optimization of prompts, systems, and skills: Prompt-only methods, Reflexion, DSPy, GEPA, and Hermes-SE optimize instructions, behavior, pipelines, or language-system components through proposal, feedback, compilation, or reflective evolution.
  • Multi-agent collaboration and agent development workflows: Multi-agent research finds benefits from role specialization, while SkillHone assigns revision proposals to optimization subagents and candidate testing to evaluation subagents.SkillHone returns redacted evidence through a permission-bounded dispatcher.

5 Conclusion

SkillHone evolves skills through role-separated subagents, redacted practice feedback, and persistent decision histories linking problems, revisions, evidence, and outcomes. It exceeds stated skill-development and deep-research baselines, improves seeded skills across seven internal scenarios, and transfers across execution backbones without further optimization.

  • 5 Conclusion: SkillHone evolves skills through role-separated subagents, redacted practice feedback, and persistent decision histories linking problems, revisions, evidence, and outcomes.These components connect problems, revisions, evidence, and outcomes across the evolution process.
  • 5 Conclusion: SkillHone exceeds iterative and reflective skill-development baselines and the deep-research agent on GAIA and WebWalkerQA-EN.The conclusion reports these benchmark comparisons without giving additional per-benchmark values.
  • 5 Conclusion: 18.8 points on average: SkillHone improves seeded skills across seven recurring internal scenarios.The reported average covers seven recurring internal scenarios.
  • 5 Conclusion: The skill bundle transfers to a different execution backbone without further optimization.This transfer result is presented as evidence accompanying the reported performance gains.

Limitations

SkillHone currently evolves only one skill at a time, leaving joint evolution of interdependent skills unaddressed.

  • Limitations: SkillHone evolves a single skill in isolation and does not address coordination across shared resources or overlapping failure modes among multiple interdependent skills.The limitation concerns joint evolution of multiple skills and their interactions.

A Existing Community Skills · B Repository Workflow Example

SkillHone uses an agent-selected pool of search-related community skills and can implement persistent decision history through a GitHub-style repository workflow. The workflow records diagnoses, revisions, and outcomes while optimizing the full skill bundle rather than only SKILL.md.

  • A Existing Community Skills: The shared existing-skill pool is selected by an agent from search-related community skills.The implementation draws this pool from ClawHub and SkillHub.
  • A Existing Community Skills: ClawHub is one source of search-related community skills in the shared pool.
  • A Existing Community Skills: SkillHub is another source of search-related community skills in the shared pool.
  • B Repository Workflow Example: SkillHone realizes persistent decision history through a GitHub-style repository workflow.This workflow is one concrete auditable interface, not a required interface for SkillHone.
  • B Repository Workflow Example: Issues record diagnosed failure modes, pull requests contain proposed skill revisions, and merge or rejection decisions record accepted outcomes.
  • B Repository Workflow Example: The repository records from one deep-research run span five non-seed SkillHone iterations and show optimization of instructions, scripts, and references.This contrasts with Hermes-SE, which rewrites only a single SKILL.md body.

C Optimization Trajectory

On identical practice probes and from the same seed skill, SkillHone improves performance while recovering from regressions through targeted follow-up edits. Compared with Hermes-SE’s whole-candidate decisions, persistent decision history enables later revisions to preserve useful edits while targeting offending changes.

  • Optimization Trajectory: SkillHone improves from 30% to 70% on the same practice probes, recovering from two regressed revisions through targeted follow-up edits.Both runs start from the same seed skill.
  • Optimization Trajectory: Hermes-SE instead accepts or skips whole prompt candidates under a scalar validation signal.This comparison uses one SkillHone run and one Hermes-SE run on identical practice probes.
  • Optimization Trajectory: Persistent decision history lets later revisions target the offending part of a change while retaining useful edits.The operational value is illustrated by revisions informed by prior decisions rather than rediscovering earlier rationale.

D Evaluation Repository and Redaction

SkillHone uses an evaluation repository of probe tasks and execution evidence to provide practice feedback while keeping optimization-side reports redacted. Final benchmark evaluation remains outside the optimization loop.

  • Repository contents: The repository stores probe instances, oracle targets, validators, provenance metadata, and execution traces for skill development.Probe items may originate from human-written cases, historical failures, or environment-derived checks.
  • Evaluation workflow: Evaluation subagents run the current skill on available probes and export redacted reports to the optimization side.This setup only assumes probe execution and report export capabilities from evaluation subagents.
  • Redaction boundary: Redacted problem reports summarize failure modes, aggregate outcomes, and diagnostic hypotheses without exposing unredacted targets, validators, or traces.For each probe, the task instance, oracle target, validator, and provenance metadata are represented as x_k, y_k, ϕ_k, and π_k.
  • Evaluation boundary: Final benchmark evaluation remains outside the optimization loop.Practice feedback is generated from repository probes rather than from the final benchmark evaluation.

E Subagent Dispatch Patterns and Permissions

SkillHone uses runtime-dispatched, permission-bounded subagents rather than fixed preconfigured roles. Team boundaries separate optimization from evaluation access, while the dispatcher creates subagents and routes artifacts without editing either repository.

  • Dispatch patterns: Runtime dispatch creates each subagent on demand, assigns it to a permission-bounded team, and grants only the actions required for its dispatch.The roles are recurring dispatch patterns observed during the development loop, not fixed identities preconfigured by SkillHone.
  • Permissions: Optimization subagents can write to the skill repository but not access unredacted probes, validators, or traces; evaluation subagents have the reverse permissions.The team boundary is the structural mechanism separating repository modification from probe and trace inspection.
  • Dispatch patterns: The runtime dispatcher creates subagents and routes artifacts but does not directly edit either repository.

F Deployment Evaluation Protocol

The deployment evaluation protocol measures recurring tool-mediated analysis tasks against human-verified reference answers. Seeded and SkillHone-optimized skills are compared on the same fixed evaluation sets under identical runtime configurations, while subagent permissions remain structurally separated.

  • Evaluation sets: Each recurring scenario uses representative de-identified requests with reference answers produced by LLM annotation and verified by human annotators.Disagreements between annotations are resolved through review.
  • Evaluation sets: Accuracy is measured by exact-match against the verified reference.
  • Subagent roles: SkillHone’s optimizer and evaluator are separated by recurring subagent dispatch patterns with distinct permission boundaries.The separation is structural rather than prompt-imposed.
  • Controlled comparison: Seeded and SkillHone-optimized skills run on the same fixed evaluation set under identical runtime configuration, isolating ∆ to skill-level changes rather than evaluation-set drift.
Loading 2606.08671v3…