Source-linked AI summary
Skills-Coach: A Self-Evolving Skill Optimizer via Training-Free GRPO
Yu Tian, Jiawei Chen, Lifan Zheng, Mingxiang Tao, Xinyi Zeng, Zhaoxia Yin, Hang Su, Xian Sun
TL;DR
LLM-agent skills are abundant but fragmented, leaving gaps for complex tasks. Skills-Coach automatically generates boundary-testing tasks, optimizes and compares skills, and evaluates them in a traceable pipeline; average scores rose from 0.37 to 0.84 and pass rates from 33.59% to 88.02%.
Problem
LLM-agent skills often target localized use cases, producing fragmented coverage and functional gaps for complex, multifaceted tasks.
Method
Skills-Coach uses automated task generation, skill optimization, comparative execution, and traceable evaluation to iteratively evolve skills without continuous human intervention.
Results
0.37 to 0.84 average score and 33.59% to 88.02% pass rate, with improvements across skill categories and task difficulties.
Takeaways & Limitations
Skills-Coach provides a closed-loop approach for autonomous skill refinement and more comprehensive skill capability coverage.
Abstract
from arXiv · showhide
We introduce Skills-Coach, a novel automated framework designed to significantly enhance the self-evolution of skills within Large Language Model (LLM)-based agents. Addressing the current fragmentation of the skill ecosystem, Skills-Coach explores the boundaries of skill capabilities, thereby facilitating the comprehensive competency coverage essential for intelligent applications. The framework comprises four core modules: a Diverse Task Generation Module that systematically creates a comprehensive test suite for various skills; a Lightweight Optimization Module dedicated to optimizing skill prompts and their corresponding code; a Comparative Execution Module facilitating the execution and evaluation of both original and optimized skills; and a Traceable Evaluation Module, which rigorously evaluates performance against specified criteria. Skills-Coach offers flexible execution options through its virtual and real modes. To validate its efficacy, we introduce Skill-X, a comprehensive benchmark dataset consisting of 48 diverse skills. Experimental results demonstrate that Skills-Coach achieves significant performance improvements in skill capability across a wide range of categories, highlighting its potential to advance the development of more robust and adaptable LLM-based agents.
1 Introduction
Skills-Coach addresses fragmented skill coverage by autonomously probing capability boundaries, optimizing skills, and evaluating their evolution in a closed loop. The framework combines four modules, supports virtual and real execution modes, and is evaluated on the 48-skill Skill-X benchmark.
- Motivation: Skills developed for localized use cases leave fragmented coverage and functional gaps in complex, specialized tasks.The skill ecosystem has abundant volume but does not systematically cover comprehensive functional requirements.
- Framework: Skills-Coach explores a target skill’s capability boundaries, analyzes optimization spaces, generates improved versions, and compiles structured reports for iterative refinement.This forms an automated closed-loop framework for Skill Self-Evolution.
- Framework modules: The framework generates comprehensive test suites, refines skill instructions and code with training-free GRPO, compares original and optimized executions, and performs traceable multidimensional evaluation.These four modules support task generation, lightweight optimization, comparative execution, and data-driven retention decisions.
- Execution modes: Skills-Coach supports virtual and real modes, with virtual mode bypassing command execution and estimating completion from criterion-related keywords and hash-derived deterministic randomness.Real mode instead executes the relevant commands or scripts.
- Evaluation: 48 skills comprise Skill-X, a diverse benchmark curated from ClawHub, Anthropic, and SkillSh for evaluating Skills-Coach across widely used skill categories.Empirical evaluations report substantial performance enhancements across a diverse spectrum of skill categories.
2 Skills Coach
Skills-Coach is an automated framework for self-evolving skills that evaluates and optimizes execution capability and robustness through a structured pipeline. Its four modules generate diverse tasks, optimize instructions and code, compare executions, and traceably evaluate results.
- Framework Overview: Skills-Coach comprises four modules: Diverse Task Generation, Lightweight Optimization, Comparative Execution, and Traceable Evaluation.Together, they support systematic task creation, skill refinement, controlled comparison, and quantitative assessment.
- Diverse Task Generation: The Diverse Task Generation Module creates training and test sets spanning standard, advanced, and boundary tasks, including edge cases and operational limits.It analyzes skill specifications to construct comprehensive evaluation suites whose quality affects optimization and performance assessment.
- Lightweight Optimization: The Lightweight Optimization Module uses Training-Free GRPO to iteratively improve skill instructions and code without conventional gradient-based parameter optimization.Instruction optimization compares variants, while code optimization applies sequential mechanisms; strategies differ for instruction-only and code-inclusive skills.
- Comparative Execution: The Comparative Execution Module runs original and optimized skills on identical tasks in controlled, isolated, reproducible environments while recording comprehensive results.Parallel execution improves efficiency, and fail-safe handling logs exceptions, preserves partial outputs, and continues processing subsequent tasks.
- Traceable Evaluation: The Traceable Evaluation Module computes normalized metrics and comparative scores from execution artifacts, using criterion parsing, task evaluation, and macro-level aggregation.It primarily uses seven-dimensional LLM assessment with evidence and falls back to enhanced heuristics when LLM evaluation is unavailable or times out.
3 Experiments
Experiments evaluate Skills-Coach on Skill-X, a 48-skill benchmark, using itemized task criteria and multiple performance metrics. Results show substantial overall and category-level improvements, with gains varying by initial skill performance.
- Benchmark: Skill-X covers 48 widely used skills from Anthropics, Clawhub, and Vercel Labs across foundational data processing and complex logical interactions.
- Evaluation: Tasks receive one point per satisfied criterion, pass at a default threshold of 70% of the maximum score, and are summarized by pass rate and score metrics.
- Experimental Setting: The default setup runs 3 optimization epochs with 12 training tasks and 8 test tasks, evenly split between standard and advanced difficulty levels.
- Overall Results: 0.37 to 0.84 average score, a 127% relative growth, and 33.59% to 88.02% pass rate, a 54.43% increase, demonstrate broad performance gains.
- Category Results: Both instruction-only and code-inclusive skills improve pass rates by over 50%, with the largest relative improvement occurring for code-inclusive skills.
- Improvement Analysis: 23 skills achieve score increases of +0.5 or higher, while five already-perfect skills remain at 1.0 score and 100% pass rate before and after optimization.
4 Conclusion
The conclusion presents Skills-Coach as an automated framework for enhancing skill self-evolution in LLM-based agents. It addresses the fragmentation of the skill ecosystem to support more comprehensive and robust intelligent applications without human intervention.
- 4 Conclusion: Skills-Coach is introduced as an automated framework that enhances self-evolution capabilities for skills in LLM-based agents.The framework is designed to empower agents with enhanced skill self-evolution capabilities.
- 4 Conclusion: The framework addresses the critical challenge of a fragmented skill ecosystem.This challenge is identified as limiting the development of more comprehensive and robust intelligent applications.
- 4 Conclusion: Skills-Coach formalizes the skill self-evolution problem and proposes an automated solution requiring no human intervention.The passage identifies this as one of the framework’s key contributions.
Appendix · A Evaluation Dimensions with 51 Discrete Evaluation Metrics
The appendix defines eight evaluation dimensions with 51 independent metrics to assess whether generated samples are functional and meet professional-grade production standards. One dimension, structural completeness and organization, evaluates documentation coverage from introduction through advanced concepts.
- A Evaluation Dimensions with 51 Discrete Evaluation Metrics: Eight evaluation dimensions comprise 51 specific assessment criteria for evaluating generated samples.The framework presents these metrics in Table 3.
- A Evaluation Dimensions with 51 Discrete Evaluation Metrics: Structural completeness and organization is one evaluation dimension containing 7 points.This dimension addresses the structure and organization of evaluated documentation.
- A Evaluation Dimensions with 51 Discrete Evaluation Metrics: The criteria require a clear introduction or overview explaining the document’s purpose and goals.The introduction should appear at the document start.
- A Evaluation Dimensions with 51 Discrete Evaluation Metrics: The criteria require complete installation and setup instructions, including environment configuration.This evaluates whether users receive the configuration information needed for setup.
- A Evaluation Dimensions with 51 Discrete Evaluation Metrics: The criteria require comprehensive usage documentation covering all commands and functions.Usage coverage is assessed as part of structural completeness and organization.
- A Evaluation Dimensions with 51 Discrete Evaluation Metrics: The criteria require at least 3 concrete examples spanning different real-world scenarios.Examples are assessed for both concreteness and scenario diversity.
- A Evaluation Dimensions with 51 Discrete Evaluation Metrics: The criteria also require configurable options, troubleshooting and FAQs, and progression from basic to advanced concepts.These requirements cover configuration, error handling, and logical instructional progression.
B Skill Sources in Skill-X
Skill-X draws its skills from multiple sources, with clawhub.ai supplying the ranked entries and GitHub contributing algorithmic-art. The listed clawhub.ai skills span ranks 1 through 11 and common-status entries.
- Skill Sources in Skill-X: clawhub.ai provides the ranked Skill-X sources, including self-improving-agent at rank 1, ontology at rank 3, weather at rank 8, and pollyreach at rank 11.Other ranked clawhub.ai entries include self-improving-proactive-agent, multi-search-engine, and polymarket-trade.
- Skill Sources in Skill-X: clawhub.ai also supplies multiple skills marked Common, including agent-browser-clawdbot, byterover, rss-daily-digest, self-evolving-skill, and stock-analysis.The source list additionally marks admapix, nano-banana-pro, and obsidian-1-0-0 as Common.
- Skill Sources in Skill-X: GitHub is listed as the source for the algorithmic-art skill.