Trajectory-Guided Fault Localization for Agent Skill Evolution

Motivation

• Expert-led refinement requires tracing unsuitable actions back to skill content and repeatedly revising and evaluating guidance, making the process costly and difficult to scale.Incomplete guidance or unsuitable workflows can impair task execution.Source: §1 · p. 1

Challenges

• Existing approaches often leave behavior-to-edit links implicit, making revisions hard to inspect and recurring problems hard to distinguish from ineffective edit locations.ContractSkill [24] explicitly localizes failing steps but assumes skills have a suitable structured form and verifier.Source: §1 · pp. 1–2
• Because agent behavior is stochastic and tasks differ, evidence from a single failure or minibatch may mistake incidental actions for shared skill deficiencies.The paper argues this can produce case-specific revisions that do not improve unseen tasks.Source: §1 · p. 2

Our Work

• Suspicious actions are traced to skill contents that may span multiple skills, with edit sites and supporting evidence recorded before revisions are generated.This action-centered localization is intended to make decisions inspectable and support coordinated revisions.Source: §1 · pp. 2–3
• SkillMorph compares failure and success evidence across repeated executions, tracks changes across evolution loops, and aggregates evidence across tasks to identify suspicious actions.An action that introduces an error later recovered from in a successful execution can count as failure evidence.Source: §1 · pp. 2–3
• 15.84% and 30.03% relative improvements in pass@3 and pass3 on SWE-Skills-Bench, and 16.67% and 100% on CannBot, are reported for SkillMorph versus the original skills.The paper also reports better overall performance than four compared skill-evolution methods and six accepted skill-revision pull requests in an AI operator-development team.Source: §1 · p. 3

2.1 Code Agent with Skill

• A code-agent trajectory consists of action-observation pairs, and a task-specific evaluation function maps the trajectory to an outcome such as pass/fail or structured correctness or performance.Agents load skill documents and supporting artifacts as needed while selecting and executing actions.Source: §2.1 · p. 3
• Skill deficiencies manifest through problematic actions because agents decide whether and how to follow natural-language instructions, and such actions may recur across related tasks.Actions provide an observable intermediate link between skill contents and outcomes.Source: §2.1 · p. 4

2.2 Skill Evolution Problem

• Skill evolution revises artifact contents over L loops to make the final skill set effective on unseen tasks, with the held-out task set never accessed during evolution.The agent model and harness are fixed, and each update changes only artifact contents.Source: §2.2 · p. 4
• Evolution should target persistent deficiencies, reuse historical evidence to preserve gains, and coordinate edits across skills whose contents jointly guide actions.Because the final skill set is returned, regressions in intermediate revisions can persist into the result.Source: §2.2 · p. 4

Example setup

• The mcp-builder package combines a workflow in SKILL.md, implementation references, and testing scripts.The example uses document-query and note-management tasks for evolution and holds out a document-indexing task.Source: §3 · pp. 4–5

Failures and localization

• On the original skill, one task fails because separate in-memory database connections cannot share imported data, while another misses shutdown-on-input-close behavior.The first task passes 11/13 checks and the second 16/17; SkillMorph traces both problematic actions to reference/node_mcp_server.md.Source: §3 · p. 5

Coordinated revision and outcome

• SkillMorph revises three sections together, adding runtime rules, matching checklist checks, and a stdio testing procedure; the evolved skill passes both illustrated tasks and the held-out task.The held-out task had the same query and shutdown failures under the original skill; with the evolved skill, retrieval succeeds and the server terminates normally.Source: §3 · p. 5

4 Approach

• SkillMorph’s three stages collect and abstract executions, localize suspicious actions, then locate edit sites and revise the skill set.An evolution memory retains execution evidence, analyses, and revision history.Source: §4 · pp. 5–6

4.1 Execution and Trajectory Abstraction

• SkillMorph runs each task repeatedly, retaining successes and failures, then normalizes verbose logs into ordered state–action–state transitions for comparison.Successful runs can contain recovered errors, while failed runs can contain effective intermediate actions.Source: §4.1 · p. 6

4.2 Suspicious Actions Localization

• It scores actions by contrasting failure and success coverage, checking errors introduced even in recovered successful runs, adjusting evidence across loops, and aggregating across tasks.Failure-coverage increases raise evidence; increases in success coverage reduce suspicion, while unchanged evidence remains neutral.Source: §4.2 · pp. 7–9

4.3 Skill Edit-Site Localization and Revision

• An LLM recursively maps suspicious actions from skills to artifacts and relevant contents, proposes revisions using execution and revision evidence, then evaluates and consolidates selected edits.A single edit site can span multiple contents across different skills or artifacts.Source: §4.3 · pp. 9–10

5.1 Experimental Setup

• The evaluation covers 49 independent SWE-Skills-Bench skills and 12 coordinated CannBot skills, with held-out task splits used to assess generalization.SWE-Skills-Bench uses complementary five-task folds; CannBot uses 15 evolution tasks and 15 held-out tasks.Source: §5.1 · pp. 10–11

5.3 RQ2: Comparison against Baselines

• SkillMorph achieves the highest pass@3, trial-level accuracy, and pass3 on both benchmarks against the original skills and four existing methods.Table 1 compares the methods across SWE-Skills-Bench and CannBot.Source: Table 1 · p. 11, §5.3 · p. 12

5.2 RQ1: Overall Performance of SkillMorph

• On held-out tasks, SkillMorph reaches 92.24% trial-level accuracy on SWE-Skills-Bench and 93.33% on all three metrics on CannBot.Across complementary SWE-Skills-Bench splits, pass@3 ranges from 93.47% to 97.55%; the framework takes about 49 minutes per evolution loop there and 127 minutes on CannBot.Source: §5.2 · pp. 11–12

5.3 RQ2: Comparison against Baselines

• Against original skills, pass@3 rises from 82.45% to 95.51% on SWE-Skills-Bench and from 80.00% to 93.33% on CannBot.Pass3 rises from 67.96% to 88.37% on SWE-Skills-Bench and from 46.67% to 93.33% on CannBot.Source: §5.3 · p. 12
• On SWE-Skills-Bench, SkillMorph has 9 regressions versus CAP-Evolve’s 36; on CannBot, it retains consistent success on all 7 originally consistent tasks.Across baselines, SkillMorph also gains more consistently solved tasks and additional successful trials; average CannBot execution time falls from 141.55 to 87.07 minutes.Source: §5.3 · pp. 12–13

5.4 RQ3: Configurations and Components of SkillMorph

• The first evolution loop yields large gains, while single-run execution most affects pass@3 and removing localization most affects pass3.Removing localization lowers pass3 by 6.12 points on SWE-Skills-Bench and 33.33 points on CannBot; removing history has the smallest effect.Source: §5.4 · pp. 13–14, Table 2 · p. 14

• The AI-operator development team accepted all 6 submitted skill-revision pull requests, covering skills used in automated kernel generation.The evaluation covered 63 operators and revisions addressing numerical correctness, kernel performance, and development and verification workflows.Source: §6 · p. 14
• For tasks passing correctness checks with both skill versions, average completion time fell from 123.63 to 72.81 minutes, a 41.11% reduction.The passage attributes the savings to workflow guidance reducing unnecessary operations, such as unfocused searches.Source: §6 · p. 14
• Across 1,788 case-level measurements, 44.18% of correct kernels matched or exceeded the reference implementation’s performance.Among these measurements, 64.26% achieved at least 0.6× speedup and 54.14% at least 0.8×.Source: §6 · p. 14

Effect of the Number of Selected Actions

• pass3 rises to 84.90% and 88.37% at n=5 and n=10, respectively, as all three metrics improve from n=1 to 5 and 10.Figure 6 varies the number of selected actions across n=1, 5, 10, 15, and 20; n=0 denotes the original skills.Source: Figure 6 · p. 15, §7 · p. 15
• At n=1, pass@3 reaches 90.00% while pass3 is 69.39%; performance remains relatively stable at n=15 and 20.The authors suggest that selecting too few actions may leave consistency-related deficiencies unaddressed, while actions beyond a certain selection size yield limited further gains.Source: §7 · p. 15

Task-Specific Adaptation

• In 21 tasks failing all three evaluation runs under both original and opposite-fold-evolved skills, five later succeeded at least once and four improved individual verification checks.These observations suggest task-specific feedback may complement cross-task experience.Source: §7 · p. 15

Skill Authoring

• Skill authors should specify applicability conditions, operation order, and completion criteria, and keep guidance consistent across instructions, related files, and scripts.The authors observed that testing instructions can still leave verification blocked when dependency handling, verification environments, or re-verification after repairs lack concrete guidance.Source: §7 · p. 15

Threats to Validity

• Validity risks include stochastic execution and approach adaptation, selected skill systems and tasks, and skill-quality metrics; mitigations include three trials, held-out cross-domain tasks, split swaps, and complementary metrics.The comparisons use official implementations under a common protocol.Source: §7 · p. 15

Agent Skill Construction and Evolution

• Skill construction derives artifacts from execution experience, while skill evolution revises existing artifacts from execution feedback; SkillMorph links suspicious-action analysis to edit-site localization and coordinated revisions.Prior evolution methods include failure-analysis loops, generated evaluation tasks, validation-gated edits, and trajectory-based defect diagnosis.Source: §8 · p. 16

Agent Trajectory Analysis and Utilization

• Trajectory research analyzes behavior, diagnoses failures, or manages context; SkillMorph instead connects behavioral evidence to editable skills and tracks revisions across evolution loops.Examples include contrasting successful and unsuccessful executions, spectrum-based fault localization for multi-agent systems, and removing redundant information.Source: §8 · p. 16

Feedback-Driven Software Improvement

• Like automated program repair, SkillMorph follows a localize-then-revise pattern, using spectrum-guided execution feedback to locate and revise deficient skill artifacts across loops.APR uses tests for fault localization and candidate-patch search and validation.Source: §8 · p. 16

Conclusion

• SkillMorph abstracts trajectories into state transitions, aggregates evidence across executions, evolution loops, and tasks, then traces suspicious actions to skill edit sites with linked rationales.Edit sites may span multiple skills.Source: §9 · p. 16
• Relative to original skills, pass@3 improves by 15.84% and pass3 by 30.03% on SWE-Skills-Bench, and by 16.67% and 100%, respectively, on CannBot, outperforming four compared approaches.The reported outcomes are relative improvements for each benchmark and metric.Source: §9 · p. 16
• In deployment with a collaboration partner, SkillMorph produced six skill-revision pull requests accepted for future AI operator development.The authors present this as evidence of practical value.Source: §9 · p. 16