PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillAxe: Sharpening LLM-Authored Agent Skills Through Evaluation-Guided Self-Refinement

Srishti Gautam, Arjun Radhakrishna, Sumit Gulwani

arXiv:2606.10546v2cs.MA

TL;DR

LLM-authored skills can fail despite fluent instructions, and coarse task-level evaluations offer little diagnostic feedback. SkillAxe diagnoses skills across four dimensions and uses those findings to guide refinement. It improves results on SkillsBench and SpreadsheetBench, including a 28% relative pass-rate gain on SkillsBench over unimproved LLM-authored skills.

  • Problem

    LLM-authored skills provide no measurable improvement over bare agents, while existing task-level pass/fail signals offer little insight into why skills fail.

    Source: §1 · p. 1

  • Method

    SkillAxe compares agent behavior with and without a skill, then uses four unsupervised diagnostics to produce structured briefs for skill refinement.

    Source: §1 · p. 2

  • Results

    Across SkillsBench and SpreadsheetBench, SkillAxe improves skill performance; on SkillsBench, pass rate rises 28% relative over unimproved LLM-authored skills.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    SkillAxe reduces the need for human expertise in skill authoring by identifying what is wrong with a draft skill and how to fix it.

    Source: §7 · p. 9

    Closing the remaining gap to human-authored skills likely requires stronger online execution feedback and multi-iteration refinement beyond the current single-update loop.

    Source: §6 · p. 9

Abstract

from arXiv · show

Skill documents, structured natural-language instructions that guide Large Language Model (LLM) agents, are critical to modern agent frameworks, yet LLMs struggle to write skills that actually work. On SkillsBench, human-authored skills improve pass rates by 16.2 percentage points, while LLM-authored skills provide no measurable gain. We introduce SkillAxe, a fully unsupervised framework that enables LLMs to iteratively diagnose and refine their own skills. SkillAxe decomposes skill quality into four interpretable dimensions (quality impact, trigger precision, instruction compliance with fault attribution, and solution-path coverage), producing structured improvement briefs that require no ground-truth labels, test suites, or environment rewards. On SkillsBench, SkillAxe improves pass rates by 28\% relative over unimproved LLM skills and closes 47--67\% of the gap to human-authored skills. We validate the approach as a continuous improvement engine in the wild on SpreadsheetBench, where a SkillAxe-built skill library learns from past agent trajectories and raises pass rate from 16.0\% to 52.0\% using only 22 skills.

1 Introduction

Skill development lacks actionable diagnostics, and LLM-authored skills provide no measurable gain over bare agents despite human-authored skills improving pass rates. SkillAxe diagnoses skill failures and refines skills, improving results on SkillsBench and SpreadsheetBench.

1 Introduction

  • LLM-authored skills provide no measurable improvement over bare agents, while human-authored skills increase SkillsBench pass rates by 16.2 percentage points.Current task-level pass/fail signals do not reveal whether failures stem from incorrect triggers, conflicting instructions, or misleading guidance.Source: §1 · p. 1
  • SpreadsheetBench captures procedural tool-use challenges across 912 real-world tasks, including formula recalculation and macro execution; existing approaches offer little guidance on improving deployed skills.Even capable agents making 10+ tool calls per task can use reusable skill guidance when it is available on demand.Source: §1 · pp. 1–2
  • SKILLAXE compares agent behavior with and without a skill, diagnosing quality impact, trigger precision, instruction compliance, and solution-path coverage to guide rewriting without task-specific labels or test suites.The introduction identifies these diagnostics as a way to provide actionable feedback for systematically refining skills.Source: §1 · p. 2
  • 41.6% vs. 32.5%: SKILLAXE raises SkillsBench pass rate 28% relative over unimproved LLM-authored skills, closing 47–67% of the gap to human-authored skills.On SpreadsheetBench, a 22-skill library raises held-out pass rate from 16.0% to 52.0%.Source: §1 · p. 2

2 Related Work

Prior work studies reusable skill libraries, evaluation, and failure attribution, but leaves limited support for refining persistent natural-language skill documents without environment-specific rewards. SKILLAXE instead uses diagnostic feedback from agent behavior to target skill rewriting.

2 Related Work

  • Reusable skill systems often store or select skills rather than refine their internal structure; iterative approaches such as Voyager [11] and SAGE [12] rely on environment-specific rewards.Other infrastructure work explores retrieving and evolving skills over time [2] [3] [14] [15] [16].Source: §2 · p. 2
  • Skill benchmarks mainly provide task-level success signals [4] [7] [8], while other approaches optimize token efficiency [17], selection [18], or prompts with supervised rewards [19] [20] [21].TextGrad and Trace [23] generate richer textual feedback, whereas Reflexion [24], Self-Refine [25], and AgentRefine [26] improve executions or model behavior rather than persistent skill documents.Source: §2 · p. 2
  • SKILLAXE attributes failures at the skill-agent boundary, distinguishing flawed instructions from agent execution errors so rewriting can target problems while preserving correct rules.Figure 1 depicts its diagnostic-and-refinement loop, in which an improvement brief guides an LLM to update a skill.Source: §2 · p. 3, Figure 1 · p. 3

3 SKILLAXE: Diagnostic Skill Evaluation and Self-Refinement

SKILLAXE compares agent responses with and without a skill, then diagnoses quality impact, triggering, instruction compliance, and solution-path coverage. These diagnostics distinguish failure sources and guide targeted refinement.

3 SKILLAXE: Diagnostic Skill Evaluation and Self-Refinement

  • SKILLAXE derives four diagnostics from the task, skill, and agent responses with and without the skill, using LLMs as judges and rewriters rather than task-specific supervised evaluators.The measures isolate whether a skill helps, activates correctly, is followed, and supports enough execution strategies.Source: §3 · p. 3

3.1 Quality Impact

  • Quality Impact compares responses with and without a skill, separating preference direction from improvement magnitude to distinguish small regressions from catastrophic failures.A negative score can prompt revision of faulty guidance rather than discarding an otherwise useful skill.Source: §3.1 · pp. 3–4

3.2 Trigger Precision

  • Trigger Precision evaluates positive and negative trigger phrases in context-aware embedding space, measuring trigger breadth, exclusion specificity, and worst-case boundary ambiguity.The figure depicts activation and rejection regions; nearby task queries expose ambiguous or over-broad triggers.Source: §3.2 · p. 4, Figure 2 · p. 4

3.3 Instruction Compliance with Fault Attribution

  • Instruction Compliance extracts skill rules and evaluates their relevance, adherence, quality, and whether failures originate from the skill or agent, grounding judgments in file-level evidence.For example, violating a precise RGB rule suggests agent failure, while violating a vague “use yellow” rule suggests insufficiently specific guidance.Source: §3.3 · p. 5
  • Fault-adjusted credit restores partial credit when failures are attributed to the agent, so low adherence alone does not mark a skill as poor.The distinction between Compliance and SkillScore prevents rewriting correct rules merely because the agent executed them poorly.Source: §3.3 · p. 5

3.4 Solution-Path Coverage

  • Solution-Path Coverage checks whether skill chunks align with plausible task strategies, since a correctly triggered, precise skill may still support only a narrow set of valid paths.Tasks may admit several approaches, such as pandas, openpyxl, Excel COM automation, or formula-driven editing.Source: §3.4 · p. 6

4 Experiments

The experiments test both task-specific skill refinement on SkillsBench and continual shared-library improvement on SpreadsheetBench. SKILLAXE improves agent performance chiefly through greater execution coverage, while its SpreadsheetBench library matches the LLM-self library’s accuracy with fewer skills.

4.1 SkillsBench Setup

  • The experiments compare task-specific SkillsBench skills with a SpreadsheetBench shared-library setting trained on 200 tasks and tested on 50.SkillsBench compares no-skill, LLM-self, one-iteration SKILLAXE, and human-authored skills under fair and native evaluation protocols.Source: §4 · p. 6, §4.1 · pp. 6–7

4.2 Multimodal Fair Grading

  • 72.7% of tasks produced evaluable output with SKILLAXE, versus 46.7% with no skill; output-task quality was 57.1% for both.The decomposition attributes the overall gain to coverage, not answer quality; skills supply procedural guidance that helps prevent execution failures.Source: Table 2 · p. 7, §4.2 · pp. 7–8
  • The fair grader evaluates task instructions, input files, and output artifacts without seeing the verifier suite, skill document, native reward, or agent trajectory.It tolerates minor formatting differences but requires concrete artifacts that substantively satisfy the task.Source: §4.2 · p. 7
  • Under fair grading, SKILLAXE achieved a 54.5% overall rate and 75.0% quality among output-producing tasks; human skills had the highest quality on both metrics.Human skills reached 75.5% native quality and 90.6% fair quality.Source: §4.2 · p. 8
  • With native SkillsBench Method D scoring, SKILLAXE achieved 27.3% mean continuous reward, versus 24.9% for no-skill, 25.8% for LLM-self, and 46.0% for human skills.The paper reports this as corroboration under native verifiers.Source: §4.2 · p. 8

4.3 Transfer to SpreadsheetBench: Skill Libraries in the Wild

  • On SpreadsheetBench, the SKILLAXE library matches LLM-self accuracy using 22 skills instead of 69.The comparison uses the common held-out test tasks (N = 50).Source: §4.3 · p. 8, Table 3 · p. 9

5 Discussion

The discussion frames skills primarily as execution knowledge and highlights deployment-time refinement and verifier bias as broader implications.

5 Discussion

  • Skills primarily raise execution reliability: coverage increases from 46.7% to 72.7%, while correctness among completed tasks remains 57.1%.The skills encode procedural knowledge such as library use, format handling, recalculation semantics, and workflow structure.Source: §5 · p. 9
  • Because diagnostics need only task instructions and agent outputs, SKILLAXE can refine skills in deployment without handcrafted test suites or environment-specific rewards.SpreadsheetBench demonstrates continual accumulation and refinement of reusable skills.Source: §5 · p. 9
  • Under fair grading, SKILLAXE scores 54.5% versus 42.7% for no-skill, suggesting native test suites can reject valid alternative strategies.The passage identifies this divergence as test-suite bias.Source: §5 · p. 9

6 Limitations

The authors identify limits in rule-level diagnosis, multi-skill evaluation, LLM judging, and the current refinement loop.

6 Limitations

  • Rule-level refinement can miss a fundamentally incorrect strategy when the skill remains internally self-consistent.Such a skill may produce few detectable rule violations.Source: §6 · p. 9
  • SKILLAXE evaluates skills independently and does not explicitly model interactions or conflicts between simultaneously active skills.Source: §6 · p. 9
  • Compliance evaluation and fair grading rely on LLM judges that may exhibit positional or calibration biases [31].The authors also state that closing the remaining gap to human-authored skills likely requires stronger online execution feedback and refinement beyond the current single-update loop.Source: §6 · p. 9

7 Conclusion

SKILLAXE is a fully unsupervised framework that diagnoses skill quality across four dimensions and produces structured improvement briefs. The conclusion presents it as a continuous improvement engine, while the broader-impact discussion notes both potential benefits and the need for review before deployment.

7 Conclusion

  • SKILLAXE decomposes skill quality into four interpretable dimensions and produces structured improvement briefs an LLM can act on.On SkillsBench, it raises pass rate by 28% relative and closes 47–67% of the gap to human skills, primarily through improved execution reliability.Source: §7 · p. 9

A Broader Impacts

  • Localized diagnostic feedback may help non-experts improve agent behavior without extensive prompt-engineering expertise.The discussion identifies procedural domains such as spreadsheet manipulation, document processing, and tool-use workflows as settings where missing execution knowledge or ambiguous instructions can cause brittle behavior.Source: A Broader Impacts · p. 13
  • Imperfect evaluations may cause automatically refined skills to propagate incorrect procedural guidance or reinforce undesirable behaviors.The risk arises from LLM-based judges and heuristic diagnostics that can misattribute failures or make flawed suggestions.Source: A Broader Impacts · p. 13
  • Automatically refined skills, especially in production or safety-critical settings, should be reviewed and validated by developers before deployment.The framework is presented as a diagnostic and refinement tool, not a replacement for human oversight.Source: A Broader Impacts · p. 13

B Per-Task Results

Table 4 lists matched-set SkillsBench tasks with the largest improvement from LLM-self to SKILLAXE under the native verifier. These examples include tasks where diagnostics identified actionable fixes to initially unhelpful skills.

B Per-Task Results

  • Court-form-filling has the largest gain, +0.50, after compliance diagnostics identified ignored field-ordering constraints.The compliance metric identified that the agent was not following field-ordering constraints in the skill.Source: B Per-Task Results · p. 13
  • Five of the eight listed tasks improve from zero reward, indicating that the initial skill provided no useful guidance.The diagnostics identified actionable fixes in these cases.Source: B Per-Task Results · p. 13
  • Table 4 covers the matched-set tasks with the largest LLM-self to SKILLAXE improvements under the native verifier.Source: Table 4 · p. 13

C Method D Native Verifier Results

Under native verifiers, one SKILLAXE iteration raises mean reward over LLM-authored skills and bare agents. The execution example and implementation show how paragraph-level replacement addresses placeholders split across XML runs.

C Method D Native Verifier Results

  • 27.3% mean reward after one SKILLAXE iteration, versus 25.8% for LLM-authored skills and 24.9% for bare agents under native verifiers.Continuous-reward results corroborate the binary pass-rate findings, though absolute values are lower because partial credit is averaged rather than thresholded.Source: C Method D Native Verifier Results · p. 13, Table 5 · p. 14

SKILLAXE skill agent (SUCCEEDS)

D Execution Example: How Skills Prevent Crashes

  • Figure 4’s offer-letter-generator case contrasts a no-skill crash with successful completion after paragraph-level replacement guidance.The no-skill agent’s per-run replacement silently fails when text is split across XML runs.Source: Figure 4 · p. 14, D Execution Example: How Skills… · p. 14

E Trigger Analysis

SKILLAXE’s trigger analysis reports improved negative-trigger specificity and wider activation discrimination. The additional visualization shows separated activation and exclusion zones, while the compliance rubric uses query-blind rule extraction followed by task-aware judging.

E Trigger Analysis

  • SKILLAXE guardrails raise negative specificity to 0.338 and boundary sharpness to 0.238, versus unimproved LLM negative specificity 0.085 (19/22 skills had zero negatives).LLM descriptions have coverage breadth of 0.41–0.44 versus 0.30 for human skills; human negative specificity is 0.006.Source: E Trigger Analysis · pp. 14–15, Table 6 · p. 14
  • Improved descriptions achieve a 3× wider discrimination margin than human skills; correct-skill similarity is 0.509 vs. 0.381, with sign-test p < 0.02 across 23 tasks.The reported increase in correct-skill similarity is +0.13 over human descriptions.Source: Table 7 · p. 15, E Trigger Analysis · p. 15

G Instruction Compliance Rubric Details

F Additional Trigger Embedding Visualization

H SpreadsheetBench Details

On the held-out SpreadsheetBench subset, both skill libraries outperform the no-skill baseline, while SKILLAXE uses fewer skills than the LLM-self library. The section describes how the libraries are built and evaluated, and how training tasks are distributed across skills.

H SpreadsheetBench Details

  • The SKILLAXE library is built from 200 training tasks and evaluated on 200 held-out test tasks drawn randomly from SpreadsheetBench’s 912 tasks.The three conditions are SKILLAXE, LLM-self, and no-skill.Source: H SpreadsheetBench Details · pp. 16–17
  • Library construction routes each task to FORK, IMPROVE, or SKIP, then uses agent trajectories for an additional improvement pass.Skills collect up to five training tasks for the first improvement phase and receive up to one further improvement using trajectory-based quality-impact and instruction-compliance metrics.Source: H SpreadsheetBench Details · p. 17
  • SpreadsheetBench evaluation recalculates output workbooks through Excel COM before comparison to golden-answer workbooks, addressing missing cached formula values.The common-subset results use the native verifier’s cell-level comparisons.Source: H SpreadsheetBench Details · p. 17
  • 52.0% vs. 16.0% (+36.0pp): both skill libraries outperform the no-skill baseline on the common test subset, with SKILLAXE using 22 skills versus LLM-self’s 69.Both libraries fix the same 18 additional tasks; SKILLAXE skills are loaded in 35.8% of queries versus 20.0% for LLM-self.Source: H SpreadsheetBench Details · p. 17

H.1 Skill Routing Distribution

  • The top five skills receive 75% of routed tasks (149/200), while six skills receive none and remain in their initial generated form.The top five cover broad patterns such as formula manipulation, VBA automation, and cross-sheet operations; untrained skills represent specialized edge cases.Source: §H.1 · p. 18, Table 11 · p. 18

I Fair Grader Protocol

  • When output files are unavailable, the fair grader defaults log extracts to incomplete unless they contain final answer values, underestimating performance in low-file-preservation conditions.The grader evaluates each run independently using the task, input files, and agent output files or rendered artifacts.Source: I Fair Grader Protocol · p. 18
Loading 2606.10546v2…