PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision

Yuxuan Liu, Zhaochen Su, Lingyun Xie, Yuhao Zhang, Qing Zong, Jiahe Guo, Zhongwei Xie, Yiyan Ji, Yauwai Yim, Hongyu Luo, Xiyu Ren, Ruan Chenyu, Haoran Li, Yangqiu Song

arXiv:2606.01139v4cs.AI

TL;DR

Imperfect LLM-authored skills are difficult to improve in cold-start settings because existing approaches depend on accumulated trajectories or direct authoring without execution-grounded refinement. SkillRevise diagnoses failures, retrieves reusable repair principles, and iteratively edits and evaluates skills; it raises SkillsBench success from 36.05% to 61.63% and shows gains across benchmarks, domains, and executors. The results support treating skills as testable procedural memory refined by observed failures, while the method remains dependent on verifier-visible feedback and additional execution budget.

  • Problem

    Existing skills may fail to improve agents, while retrieval and self-evolution face task-mismatch or cold-start limitations and direct authoring lacks execution-grounded refinement.

  • Method

    SkillRevise diagnoses verifier-facing failures, retrieves repair principles, applies execution-anchored edits, re-executes candidates, and selects the best observed skill by utility.

  • Results

    61.63% SkillsBench success improves GPT-5.5 from 36.05% without skills and 39.53% with one-shot skill generation, with consistent gains across evaluated benchmarks and executor models.

  • Takeaways & Limitations

    Reusable skills are most effective when treated as testable procedural memory refined by observed failures rather than static advice.

  • Takeaways & Limitations

    SkillRevise depends on verifier-visible feedback and additional execution budget; sparse or misaligned tests can cause overfitting, and long-running or safety-critical deployments are not evaluated.

Abstract

from arXiv · show

Agent skills are procedural artifacts that enable LLM agents to execute workflows, verify constraints, and recover from failures. Existing self-evolving methods refine skills using accumulated trajectories. However, they struggle in cold-start settings, where only an initial, imperfect skill is available. Consequently, skill construction defaults to expert authoring or one-shot LLM generation. Expert-authored skills are costly and may not align with how LLM agents actually execute tasks, while one-shot generated skills can be syntactically well formed yet behaviorally weak. To bridge this gap, we propose SkillRevise, an execution-grounded framework designed to iteratively refine these initial skills. SkillRevise diagnoses skill defects from execution evidence, retrieves relevant repair principles from a general memory, and applies execution-anchored edits. By re-executing candidates and measuring empirical utility, it retains the best observed skill within the revision budget. Evaluated across three main benchmarks, two domain-specific studies, and six LLMs, SkillRevise substantially outperforms one-shot baselines, improving the base agent's success rate on SkillsBench from 36.05% to 61.63%. Furthermore, the revised skills transfer across both executors and task environments, suggesting that SkillRevise captures reusable procedural knowledge beyond any single executor. Our code is available at https://github.com/HKUST-KnowComp/skillrevise.

1 Introduction

SkillRevise addresses the cold-start problem of improving imperfect LLM-authored skills by using execution evidence to diagnose defects, retrieve repair principles, and revise skills within a bounded evaluation loop. Across benchmarks and executor models, the framework improves skill performance over no-skill and one-shot baselines.

  • Motivation: Existing skills may provide little benefit or degrade performance when poorly matched to the executor, task context, or environment.This motivates shifting attention from skill adoption to skill acquisition.
  • Motivation: Retrieval and self-evolution methods face task mismatch or cold-start limitations, while direct authoring can produce skills without execution-grounded validation.The paper positions SkillRevise as a bounded alternative starting from an existing LLM-authored skill.
  • Approach: SkillRevise diagnoses task-specific failures, retrieves reusable repair principles, applies execution-anchored edits, re-executes candidates, and selects the best observed skill by utility.Its revision loop couples current execution evidence with general repair knowledge.
  • Results: 61.63% success on SkillsBench improves GPT-5.5 from 36.05% without skills and 39.53% with one-shot skill generation within three revision rounds.The evaluation uses a unified verifier-driven harness and extends beyond the original benchmark distribution.
  • Results: Across SkillsBench, SkillLearnBench-Random, SWE-Skills-Bench-Hard, ALFWorld, and BFCL-v4, execution-grounded revision consistently improves performance across multiple executor models.The reported studies include standard, out-of-distribution, and domain-specific settings.

2 Related Work

Related work treats agent skills as reusable procedural artifacts and studies retrieval, continual skill generation, and skill injection across task settings. SkillRevise instead revises a reusable skill artifact and validates its downstream effect under a fixed verifier.

  • Agent skills and benchmarks: Agent skills extend LLM agents beyond one-off prompts and atomic tools, while SkillsBench finds curated skills helpful but self-generated skills unreliable.These findings motivate evaluating how skills are acquired and improved.
  • Agent skills and benchmarks: SkillLearnBench studies accumulating task experience into reusable skills, WildSkills studies retrieval from large collections, and SWE-Skills-Bench evaluates skill injection in software repositories.Together, these benchmarks emphasize alignment with the executor, task context, and verifier.
  • Execution-grounded revision: Unlike methods that improve a single trajectory or final answer, SkillRevise revises a reusable skill artifact and validates its downstream effect under a fixed verifier.The distinction is between artifact-level revision and trajectory- or answer-level improvement.

3 Method: SKILLREVISE

SKILLREVISE is a bounded, execution-grounded loop that diagnoses failures in an existing skill, retrieves and binds reusable repair principles, applies anchored edits, and re-executes candidates. It returns the highest-utility observed skill rather than necessarily the latest revision, while using verification and preservation constraints to protect successful behavior.

  • The episode executes the current skill, diagnoses verifier-facing failures, retrieves and binds repair principles, edits with executable anchors, re-executes candidates, and applies utility-gated selection.The loop begins from an initial skill and maintains a bounded revision process.
  • Diagnosis: Diagnosis combines verification specifications, failure attribution, and preservation constraints to identify what to repair, justify the repair, and preserve already successful behavior.Verification covers observable requirements; attribution summarizes failed checks, behavior, probable causes, and defect labels.
  • Principle Memory: Principle Memory stores reusable repair patterns rather than task solutions, while binding retains only principles whose evidence and transfer constraints fit the current diagnosis.The seed library contains seven manually consolidated, domain-general principles and is frozen before evaluation.
  • Revision Operator: Execution anchors connect textual edits to executor behavior by specifying the action, evidence or artifact to inspect, and instruction location.An anchored JSON repair can require reloading and parsing a file, checking required keys, and running local validation.
  • Utility-Gated Selection: The returned artifact is the best observed skill under the revision budget, not necessarily the latest generated candidate, and increasing the budget only expands the evaluated set.The current skill may advance for later diagnoses, while the returned skill remains utility-gated.
  • Revision Episode: Each round records trajectories, verifier feedback, outcome scores, and execution costs, then revises only when the diagnosis supports a skill-level defect.When no supported principle exists, the system can use diagnosis alone or abstain from editing; candidates are re-executed before selection.
  • Utility-Gated Selection: Utility can combine outcome, success-conditioned efficiency, transfer, and interference cost, with efficiency credit gated on verifier success.For binary verifiers, lower-cost candidates receive efficiency credit only after satisfying the verifier, preventing early stopping or omitted work from appearing efficient.
  • Memory Absorption: Optional memory absorption abstracts validated repair patterns after evaluation, but benchmark results use banks fixed before evaluation without writing back evaluation-task feedback.Absorption filters require evidence-backed, utility-improving, transferable principles and remove task-specific shortcuts.

4 Experiments

SKILLREVISE is evaluated under controlled, verifier-driven conditions across three benchmarks, multiple executors, and domain-specific studies. Across these settings, budgeted execution-grounded revision generally outperforms initial and one-shot skills, with early rounds providing substantial recovery and later rounds expanding candidate coverage.

  • Experimental Setup: Three verifier-driven benchmarks use a three-round revision budget, with GPT-5.5 authoring and evaluation across five executor models.The benchmarks are SkillsBench, SkillLearnBench-Random, and SWE-Skills-Bench-Hard.
  • Experimental Setup: Controlled comparisons isolate installed skill artifacts and revision budgets under the same task interface, executor, workspace, and verifier.Conditions include no skill, one-shot Skill-Creator, initial v0, Revision v1, and Revision v3; final selection is utility-gated over evaluated history.
  • Main Results: Revision v3 improves overall success over no-skill execution across all five executors, including GPT-5.5 from 79 to 115 successes across 206 tasks.The corresponding totals are 55 to 100 for Opus-4.7, 47 to 86 for Kimi-2.6, 33 to 77 for Qwen-3.6-Plus, and 49 to 95 for DeepSeek-V4-Pro.
  • Main Results: One-shot generation is inconsistent: GPT-5.5 reaches 78/206 versus 79/206 without skills, while Opus-4.7 falls from 55/206 without skills to 47/206 with v0.Revision v3 is the strongest overall condition for every executor in Table 1.
  • Revision Dynamics: Revision v1 improves over v0 in every model–benchmark setting, while later rounds add verifier-tested candidates for unresolved tasks.On SkillsBench, GPT-5.5 rises from 40.7% to 54.7%; Opus-4.7 rises from 37.2% to 51.2% across later rounds.
  • Baseline Comparison: On GPT-5.5 SkillsBench, SKILLREVISE reaches 53/86 using the same 36.6k deployment tokens as direct no-skill execution, versus 42/86 for SkillHone and 46/86 for SkillOpt.The comparison uses the same 86-task GPT-5.5/Codex evaluation and aligned maxrev3 budget.

5 Analysis and Discussion

The analysis examines transferability, revision-budget selection, component contributions, and domain-specific calibration. Across these settings, execution-grounded revision improves skills, while diagnosis and structured evidence use are especially important.

  • 5.1 Analysis of Skills Transferability: Fixed GPT-5.5-produced skills improve over no-skill execution across four target executors, but executor-specific Revision v3 is stronger in every case.The transfer comparison uses a 57-task GPT-5.5 source-success subset.
  • 5.2 Revision Rounds and Selection: The utility gate retains early skill versions frequently, while later revision rounds add bounded search opportunities for unresolved tasks.Under maxrev5, v0 is retained for 37 tasks and v1 for 20, whereas v4 and v5 form only a small tail.
  • 5.3 Component Ablations: Removing Diagnosis drops success from 53/86 to 28/86, making it the dominant component in the matched ablation.Removing the preserve ledger, execution anchors, or Principle Memory also causes smaller but substantial drops.
  • 5.3 Component Ablations: Free-form repeated revision reaches 33/86 after three rounds, versus 53/86 for full SKILLREVISE, showing that additional feedback and retries alone do not explain the gains.The unstructured control starts from the same v0 result of 35/86.
  • 5.4 Domain-Specific Studies: Calibration-built principles support transfer beyond file tasks: the selected run solves 71/100 ALFWorld tasks and reaches 91/100 single-turn and 59/100 multi-turn BFCL-v4 success.In BFCL-v4 multi-turn base, revision improves over v0 by more than 30 points.

6 Conclusion

The paper presents SKILLREVISE as a bounded loop that diagnoses, revises, and selects skills using verifier feedback. Results across benchmarks and domain studies support execution-grounded skills as testable procedural memory refined by observed failures.

  • 6 Conclusion: SKILLREVISE improves over no-skill execution and one-shot Skill-Creator in most evaluated settings across three main benchmarks.Domain studies further show that frozen, calibration-acquired principles remain useful during evaluation.
  • 6 Conclusion: The results characterize effective skills as testable procedural memory refined by observed failures rather than static advice.This conclusion is stated across the benchmark and domain-specific evaluations.

Limitations

The method depends on verifier-visible feedback and additional execution budget, and its evaluation scope excludes long-running or safety-critical deployments. The benchmark setup also uses adapted interfaces and frozen skill configurations.

  • Limitations: Sparse, flaky, or misaligned tests can cause revision to overfit visible checks.The limitation follows from SKILLREVISE's dependence on verifier-visible feedback.
  • Limitations: Some tasks remain better solved by direct no-skill exploration because maxrev3 does not fall back to no-skill execution.The stated revision budget therefore does not cover every task preference.
  • Limitations: The evaluation does not cover long-running or safety-critical deployments.The paper identifies this as an explicit scope boundary.
  • Evaluation Scope: The three main benchmarks are converted to a shared SkillsBench-style interface before evaluation, while ALFWorld remains a separate interactive study.The adapted benchmarks preserve selected task assets and verifier contracts in the shared layout.
  • Evaluation Setup: SKILLREVISE starts from an initially authored v0 skill and uses a fixed Principle Memory with up to three revision rounds in the main benchmark setup.Skill-Creator performs no execution-grounded revision, whereas SKILLREVISE evaluates and revises v0.

B.2 GPT-5.5 SkillsBench Execution Profile

The GPT-5.5 SkillsBench profile uses verifier-scored execution, bounded revision, and utility-gated selection. Revision adds measurable repair cost, while the standard episode records evidence and candidate decisions for reproducibility.

  • Execution Profile: GPT-5.5 authors, revises, and executes skills under a common verifier, with SKILLREVISE attempting up to B = 3 revisions after evaluating v0.The executor is the Codex ACP agent under the Docker BenchFlow backend.
  • Cost Profile: Revision generation adds 23.1, 40.6, and 55.5 seconds per task for maxrev1, maxrev2, and maxrev3, respectively.At maxrev3, revision generation accounts for 10.3% of combined repair-stage latency.
  • Budget Trade-offs: Maxrev1 captures 66.7% of the maxrev3 gain using 42.0% of its repair-execution tokens and 42.1% of its total repair-stage latency.Maxrev2 captures 88.9% of the gain using 72.8% of tokens and 74.4% of latency.
  • Revision Episode: The standard episode evaluates v0, constructs diagnosis from current evidence, retrieves and binds principles, revises with anchors, and selects the best observed skill under utility.Terminal success stops further candidate generation in the reported benchmark runs.
  • Trace Logging: Each revision trace records the verifier contract, failure and preserve ledgers, selected and ignored principles, execution anchors, and acceptance signals.Runtime logging additionally captures status, reward, timeouts, tokens, tool calls, steps, and latency.

D Cross-Model Transfer Details

Transfer tests show that a fixed GPT-5.5-produced skill improves no-skill execution across all four target executors, but executor-specific revision remains stronger and transfer gains vary.

  • Transfer results: All four target executors improve over no-skill execution when using the fixed GPT-5.5-produced skill.The transfer runs use the same 57-task GPT-5.5 source-success subset.
  • Transfer results: +17 successes is the largest transfer gain for Opus-4.7, followed by +13 for DeepSeek-V4-Pro, +5 for Qwen-3.6-Plus, and +3 for Kimi-2.6.The gain size varies substantially across target executors.
  • Transfer results: Executor-specific Revision v3 exceeds the transferred GPT-5.5 skill in every target-executor row.This pattern indicates that transfer preserves reusable guidance but does not match executor-specific revision.
  • Interpretation: The transfer pattern suggests revised skills combine reusable task-execution paths with executor-specific prompt conventions.Utility depends on how well the target executor follows the source skill’s tool-use and verification conventions.

E Domain-Level SkillsBench Breakdown

Revision v3 improves SkillsBench performance across most domains for GPT-5.5 and Opus-4.7, but gains are not universal and some domain-level regressions remain.

  • GPT-5.5: GPT-5.5 Revision v3 improves over no-skill execution in 9 of 11 SkillsBench domains, raising aggregate success from 31 to 53.The largest absolute increases are Software, from 3/11 to 8/11, and Robotics, from 2/6 to 6/6.
  • GPT-5.5: GPT-5.5 gains are also concentrated in Software and Robotics, followed by Office and Math with +3 successes each.These improvements show that the aggregate gain is not concentrated in a single task family.
  • GPT-5.5: GPT-5.5 performance drops in Media from 5/11 without skills to 4/11 after revision.This is the main reported GPT-5.5 domain exception.
  • Opus-4.7: Opus-4.7 Revision v3 improves over no-skill execution in 9 of 11 domains, with largest gains in Office, Science, Media, and Software.The Robotics row is a counterexample: Skill-Creator reaches 5/6 while Revision v3 reaches 3/6.

F Case Studies

Case studies show revision recovering verifier failures and repairing skill-induced errors through operational safeguards, while also exposing regressions, cost trade-offs, and retention constraints.

  • Recovered failures: SKILLREVISE’s selected skill can recover failures by adding evidence-based checkpoints, targeted fixes, validation, and fallback handling.In the web-performance case, v0 fails the 800ms checkout-latency threshold while v1 succeeds after reframing the skill as a non-regression workflow.
  • Recovered failures: In the spreadsheet case, revision succeeds by snapshotting workbook cells, identifying the exact target, editing only the non-formula cell, and comparing the reopened output.The initial failures included corruption of an unrelated CNY-to-EUR cell from 0.127 to NaN.
  • Residual regressions: Skill conditioning can harm tasks that direct exploration already solves: the video-filler case remains a failure because the generic workflow over-regularizes segment detection.No revised candidate succeeds, and maxrev3 does not fall back to no-skill execution.
  • Comparative outcomes: Against Skill-Creator, maxrev3 wins on success for 20 tasks, ties on 65, and loses on 1, while overall utility wins are 58, ties 2, and losses 26.The gap reflects successful revisions that increase execution cost, along with attribution, overfitting, and missing-domain-method failures.
  • Failure analysis: Diagnosis labels across 86 GPT-5.5 SkillsBench tasks are dominated by context pollution and false certainty, appearing in 207 and 181 candidate-level entries respectively.Over-specificity and wrong abstraction level each appear in 54 entries, while environment mismatch appears once.
  • Retention constraints: The leakage audit distinguishes current-episode evidence from retained reusable guidance, requiring retained skills to avoid task-instance answers, hidden verifier facts, and one-off shortcuts.Current task instructions and traces may guide revision, but retained artifacts should encode discovery and validation rather than memorized constants.

J Additional Limitations

The paper’s additional limitations concern verifier reliability, revision cost, evaluation scope, and the absence of lifelong memory management, with several proposed extensions left untested.

  • Verifier dependence: Shared errors in Diagnosis and Utility can survive verifier dependence, especially with flaky or partial validation.The reported experiments use reproducible benchmark verifiers and do not empirically evaluate repeated evaluation, stability penalties, or complementary holdout checks.
  • Revision cost and stopping: Revision trades additional model calls, tool use, and wall-clock time for better skills, while fixed budgets may miss easy-task stopping points or specialized domain methods.Adaptive stopping, cheaper diagnosis, and earlier detection of misleading skills remain future priorities.
  • Scope and generalization: The evaluation centers on tasks with explicit, reproducible success criteria and does not establish behavior in long-running, safety-critical, adversarial, or self-evaluating environments.AgentVista and Tool Decathlon are outside scope, and maxrev3 lacks fallback to no-skill execution.
  • Memory maintenance: The experiments evaluate fixed principle banks and calibration-then-freeze updates rather than lifelong memory management.Continuous deployment would require consolidation, conflict resolution, pruning, and stability checks across repeated updates.
  • Implementation boundary: The implementation uses shared method-level authoring and revision templates across three benchmarks, with task-specific placeholder contents varying by benchmark and instance.Platform API configuration, credentials, and provider-specific hidden system prompts are omitted.

L.1 Principle-Guided Repair Contrasts

The contrasts show that principle-guided revisions turn verifier feedback into reusable repair procedures rather than instance-specific patches. These procedures improve recovery by preserving valid work, validating serialized artifacts, and narrowing skills to tasks where they affect execution.

  • Travel planning: Travel planning succeeds when missing evidence triggers a bounded fallback that preserves already-passing itinerary constraints instead of guessing or replanning.The standard run selects v3 and succeeds, whereas the no-principles run selects v1 and fails.
  • Manufacturing maintenance: Manufacturing maintenance benefits when repairs discover the local contract and express computation fixes as reusable task-family procedures rather than memorized instance patches.The standard run succeeds with v1, while no-principles revisions continue failing on the same method-level defect.
  • Grid dispatch: Checkpointed workflows catch objective or serialization drift by validating the emitted report and recomputing task-critical quantities after writing.The standard grid-dispatch run succeeds, while the no-principles candidate fails despite producing a report.
  • Grid dispatch: Narrow execution triggers reduce distraction by activating guidance only for structured DC-dispatch report tasks and excluding unrelated domain analysis.This focuses executor budget on contract discovery, optimization, serialized-report checks, and bounded recovery.
  • Cross-contrast synthesis: Across the contrasts, principles convert verifier feedback into reusable repair operations that help standard runs recover failures the no-principles ablation continues to rewrite around.The comparison emphasizes procedural repair rather than task answers.
Loading 2606.01139v4…