Skills-Coach: A Self-Evolving Skill Optimizer via Training-Free GRPO

• Individually developed skills often focus on localized use cases, producing abundant but fragmented coverage that leaves gaps and integration bottlenecks in complex tasks.The paper argues that this fragmentation hinders robust, scalable deployment of LLM-based agents.Source: §1 · p. 1
• The framework targets three challenges: generating systematic boundary-probing tasks, evolving skills from failures without conflicts, and evaluating performance, generalization, and collaboration.Boundary tasks should be challenging and comprehensive; evaluation should provide reliable feedback to guide evolution.Source: §1 · p. 2
• Skills-Coach explores skill boundaries, analyzes optimization opportunities, generates improved versions, and supports iterative refinement without continuous human intervention [13].The framework is designed to help developers refine skills and compile findings into structured reports.Source: §1 · p. 2
• Its four modules generate standard and edge-case tests, optimize skill instructions and code with training-free GRPO, compare executions, and evaluate versions using multidimensional criteria.The evaluation module calculates metrics and supports data-driven retention decisions.Source: §1 · pp. 2–3
• Skills-Coach offers virtual evaluation using instruction keywords and deterministic hash-based random values, or real-mode evaluation using outputs, logs, and errors from actual execution.Source: §1 · p. 3
• Skill-X comprises 48 widely used skills curated from ClawHub, Anthropic, and SkillSh, and evaluations report substantial performance enhancements across diverse skill categories.The authors present the benchmark as a resource for evaluating the framework and advancing robust, adaptable LLM agents.Source: §1 · p. 3

2.1 Overview of Skills Coach

• Skills-Coach generates training and test tasks, optimizes skill instructions and code, compares original and optimized skills, and produces analytical evaluation reports.Its task sets cover standard use cases and challenging edge cases; comparative execution supplies results for subsequent evaluation and iterative optimization.Source: §2.1 · pp. 3–4

2.2 Diverse Task Generation Module

• The Diverse Task Generation Module creates diverse, realistic tasks from standard operations to boundary cases, with distinct test contexts and automated validation criteria.It parses skill specifications to identify capabilities and failure scenarios. The framework also specifies four test-suite quality principles and eight evaluation dimensions comprising 51 metrics.Source: §2.2 · p. 4

2.3 Lightweight Optimization Module

• The Lightweight Optimization Module uses Training-Free GRPO[14–16] and parallel instruction and code pathways to iteratively refine skills.Code optimization combines rule-driven changes, LLM-based command refinement, and failure-based auto-fixes; strategies differ for instruction-only and code-inclusive skills. The paper reports reducing training time from hours to minutes and data needs from thousands of samples to dozens.Source: §2.3 · pp. 4–5

2.4 Comparative Execution Module

• The Comparative Execution Module runs original and optimized skills under identical, isolated conditions and logs outputs, errors, side effects, and performance metrics without scoring them.It uses temporary workspaces, sequential execution, cleanup, and optional parallel processing; its fail-safe logging preserves partial outputs and supports reproducible failure analysis.Source: §2.4 · p. 5

2.5 Traceable Evaluation Module

• The Traceable Evaluation Module scores execution results using consistent criteria, aggregates metrics, and makes retain-or-discard decisions with detailed justifications.LLM assessment scores seven dimensions from 0–100 with supporting evidence Zheng et al. [23]; an enhanced heuristic mode provides a fallback if the LLM is unavailable or times out.Source: §2.5 · pp. 5–6

3.1 Setup

• Skill-X covers 48 widely used skills from Anthropics, Clawhub, and Vercel Labs, split into 29 instruction-only and 19 code-inclusive skills.The benchmark assesses comprehension, orchestration, execution, and optimization across varied interfaces and complex task requirements.Source: §3.1 · p. 6
• Tasks receive one point per satisfied criterion, with passing set at 70% of the maximum score by default; reported metrics are pass rate and average, standard, and advanced task scores.Instruction-only skills are assessed on eight dimensions, while code-inclusive skills are evaluated on execution outcomes, valid outputs, data matching, and complex-scenario robustness.Source: §3.1 · pp. 6–7
• By default, each skill receives three optimization epochs, 12 training tasks, eight test tasks, three rollout variants per epoch, and up to two Auto-Fix rounds.Execution defaults to real mode, and claude-sonnet-4-6 is the default model for LLM-powered optimization and fixing, with heuristic fallbacks when API generation is unavailable.Source: §3.1 · p. 8

3.2 Main Result

• Average score rose from 0.37 to 0.84 (127% relative growth), while pass rate rose from 33.59% to 88.02% (+54.43%) on Skill-X.The paper reports significant improvements across all evaluation metrics.Source: §3.2 · p. 8
• Both instruction-only and code-inclusive skills gained over 50% in pass rate, while advanced tasks improved more than standard tasks.The largest relative improvement was for code-inclusive skills, which the paper associates with tasks demanding complex logical reasoning.Source: §3.2 · p. 8

3.3 Further Analyses

• In the Exceptional Improvement category, 23 skills gained at least +0.5 in score; four, including Browser and Ontology, rose from 0.0 to 1.0.Their pass rates rose from 0% or lower levels to 100% or nearly 100%. Table 2 analyzes optimization results across all 48 skills.Source: Table 2 · p. 7, §3.3 · p. 8
• Five skills already at 1.0 score and 100% pass rate stayed perfect, while react-view-transitions improved only +0.08, from 0.92 to 1.0.The analysis recommends prioritizing skills with low intrinsic performance and room to improve, since optimizing near-perfect skills yields negligible returns.Source: §3.3 · p. 8

3.4 Case Study

• The Pollyreach case study contrasts standard command/output checks with advanced same-day delivery, JSON, and parameter-validation tasks, then reports per-task and capability-boundary comparisons across four metrics.The report compares pass rate, average score, standard task score, and advanced task score between original and optimized skills.Source: §3.4 · pp. 8–10, Figure 3 · p. 9, Figure 4 · p. 9

4 Conclusion

• Evaluation on Skill-X’s 48 diverse skills shows substantial improvements across skill categories and task complexities, with especially pronounced gains for code-inclusive skills and advanced tasks.The passage describes these results as demonstrating the framework’s effectiveness through rigorous evaluation.Source: §4 · p. 10
• Training-Free GRPO offers a competitive alternative to parameter optimization, achieving results in minutes rather than hours while dramatically reducing data requirements.The passage presents this as an efficient optimization approach.Source: §4 · p. 10
• Skills-Coach formalizes skill self-evolution as an automated solution requiring no human intervention, reducing burdens on skill developers and maintainers.The framework is designed to address fragmentation in the skill ecosystem and support more comprehensive, robust intelligent applications.Source: §4 · p. 10

A Evaluation Dimensions with 51 Discrete Evaluation Metrics

• Eight evaluation dimensions comprise 51 independent assessment metrics for the generated samples.Table 3 identifies the scope as eight evaluation dimensions and 51 specific assessment criteria.Source: A Evaluation Dimensions with 51… · p. 12, Table 3 · p. 12
• The evaluation is intended to ensure samples are functional and meet professional-grade standards suitable for production environments.Source: A Evaluation Dimensions with 51… · p. 12

B Skill Sources in Skill-X

• The 48-skill inventory comprises 15 clawhub.ai skills, 16 from Anthropic’s skills repository, 8 from Vercel’s agent-skills repository, and 9 from skills.sh.These counts are tallied from the listed entries.Source: B Skill Sources in Skill-X · p. 13
• Among the ranked entries, self-improving-agent and find-skills rank first in clawhub.ai and skills.sh, respectively.Other listed ranks include ontology (3) and self-improving-proactive-agent (4) on clawhub.ai, and remotion-best-practices (5) on skills.sh.Source: B Skill Sources in Skill-X · p. 13