Source-linked AI summary
SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use
Jiayin Zhu, Kelong Mao, Yudong Guo, Dengbo He, Sulong Xu, Simiu Gu, Yutao Yue
TL;DR
Reliable skill-use remains difficult to assess because final success can mask flawed process. SkillCoach addresses this with self-evolving, skill-grounded rubrics, which improve evaluation and training supervision over outcome-only filtering.
Problem
Existing work has not used skill structure to define process supervision for evaluating and training agentic skill-use.
Method
SkillCoach derives self-evolving, skill-grounded process rubrics from real rollouts to assess selection, following, composition, and reflection.
Results
Evolved-rubric filtering improves Qwen3.5-4B from 8.0 to 24.0 and Qwen3.5-9B from 14.0 to 32.0, outperforming outcome-only filtering.
Takeaways & Limitations
Rubric evolution provides stronger supervision for selecting reusable skill-use demonstrations than verifier-passing outcomes alone.
Takeaways & Limitations
Experiments use a smaller selected task set than large production skill repositories and report only offline supervised fine-tuning, not reinforcement learning or deployment feedback.
Abstract
from arXiv · showhide
Skills are becoming a reusable operational layer for LLM agents, encoding SOPs, domain rules, tool workflows, scripts, and validation routines. In realistic skill repositories, overlapping skills make reliable skill-use difficult. Final verifier success is too coarse for both evaluation and training, since an agent may pass through trial and error while selecting distractor skills, skipping required steps, composing workflows incorrectly or omitting final checks. We introduce SkillCoach, a self-evolving rubric framework for evaluating and enhancing agentic skill-use. SkillCoach derives skill-grounded process rubrics from real rollouts and evaluates trajectories along four dimensions: skill selection, skill following, skill composition, and skill-grounded reflection. It keeps the external verifier as a separate outcome signal, allowing process quality to be distinguished from accidental task success. The evolved rubrics further serve as process supervision for selecting high-quality training trajectories. Experiments show that evolved rubrics substantially improve evaluation quality, expose failures hidden by final accuracy, and provide stronger supervision signals than outcome-only filtering for enhancing agentic skill-use.
1 Introduction
Agentic skills package reusable operational knowledge, but overlapping repositories make reliable use difficult and final task success an incomplete evaluation signal. SkillCoach addresses this gap with self-evolving, skill-grounded process rubrics that diagnose and improve skill-use beyond verifier outcomes.
- Final verifier success can hide distractor selection, skipped steps, trial-and-error behavior, and omitted skill-specified checks.Verifier-passing trajectories are therefore not necessarily reusable demonstrations for training.
- SkillCoach evaluates agentic skill-use across selection, following, composition, and skill-grounded reflection.The external verifier remains a separate outcome signal, distinguishing reliable process behavior from accidental task success.
- SkillCoach targets skill-dependent tasks in distractor-augmented libraries that approximate enterprise repositories with overlapping workflows.
- The evolved rubrics both diagnose skill-use failures and filter verifier-passing trajectories into higher-quality supervised-training demonstrations.
- SkillCoach evolves task-level rubrics from real rollouts through evidence-grounded judging, arbitration patches, and validation-gated updates.
2 Related Work
Related work studies skill artifacts, skill libraries, lifecycle management, and trajectory-level evaluation. SkillCoach differs by grounding process supervision in the structure of concrete skills and their dependencies.
- Skill representations package procedural knowledge, applicability conditions, workflows, and optional executable resources for LLM agents.
- Skill benchmarks evaluate curated, self-generated, and continually generated skills across trajectory behavior and final outcomes.
- Other approaches improve skill artifacts or consuming policies through refinement, synthesis, validation, optimization, and lifecycle management.
- Trajectory-level and rubric-based evaluators provide process assessment and actionable feedback beyond final-answer accuracy.
- Existing criteria are not jointly grounded in gold skills, distractors, required steps, cross-skill dependencies, and skill-grounded reflection.
3 Method
SkillCoach evaluates skill-dependent tasks with distractor-augmented libraries, scoring observable trajectories rather than relying on verifier success alone. It evolves evidence-grounded rubrics across four process dimensions and uses validation-gated patches to improve trajectory judging.
- Framework: SkillCoach evaluates rollouts along skill selection, skill following, skill composition, and skill-grounded reflection while keeping the external verifier separate.This separates reliable skill-use behavior from accidental verifier passing.
- Task construction: Tasks are filtered for skill dependence by comparing no-skill and gold-skill success rates and measuring key-step coverage.The selected tasks show lower no-skill success and substantially larger gains from gold skills than the original pool.
- Task construction: Distractor-augmented libraries test whether agents can avoid plausible but irrelevant skills in overlapping repositories.The default setting pairs each task with unrelated and additional distractor skills, approximating a shared enterprise skill pool.
- Process dimensions: Skill following requires visible evidence for completing weighted key steps, so unsupported claims and skipped script executions reduce credit.Each step receives an observed completion level and an evidence multiplier.
- Process dimensions: Skill composition scores valid precedence ordering and transfer of required intermediate artifacts, while single-skill tasks are excluded from this dimension.For dependencies (u, v), the trajectory must complete u before v and transfer the required state or output.
- Rubric evolution: Rubrics evolve from real rollouts through evidence-grounded judging, targeted arbitration patches, and validation-gated acceptance.The rubric defines scoring rules, evidence requirements, and negative cases with stable identifiers for local patching.
4 Experiments
SkillCoach experiments separate process quality from final task success, showing that evolved rubrics improve evaluation, diagnose distractor-induced failures, and select better training trajectories. Scaling tests further reveal distinct degradation and collapse regimes as skill libraries grow.
- Rubric validation: 83.70 versus 71.56 gold-keypoint coverage shows that Rbest captures more human-gold process evidence than R0.Usability rises from 81.53 to 94.33, while hallucination rate falls from 2.00 to 0.00.
- Rubric validation: 96.00 versus 82.00 trajectory-filtering consistency indicates stronger alignment between evolved-rubric decisions and human-gold trajectory judgments.The audit uses paired instances, macro-averaging, an independent judge, and excludes verifier outcomes.
- Skill-library performance: 18.0 to 88.0 final accuracy for Opus 4.7 and 24.0 to 80.0 for GPT-5.5 confirms that these tasks depend on reusable skills.Adding distractors still reduces performance, indicating that reliable selection and execution matter beyond skill availability.
- Skill-library performance: Qwen3.5-9B selection drops from 92.0 to 44.0 under distractors, while Gemini 3.1 Pro falls from 98.0 to 78.0.The four dimensions distinguish wrong skill choice from incomplete following, weak composition, or missing reflection.
- Rubric-filtered training: 24.0 for Qwen3.5-4B and 32.0 for Qwen3.5-9B follow Rbest-filtered SFT, outperforming both initial-rubric and outcome-only filtering.Outcome-only SFT reduces the 4B result from 8.0 to 6.0 and raises 9B only from 14.0 to 18.0.
- Distractor-boundary analysis: 6.4k–6.5k, 20k–21.25k, and 35k–35.5k distractors mark collapse boundaries for DeepSeek V4 Flash, Kimi K2.6, and Gemini 3.1 Pro.At these points, no-gold-selected and complete-failure rates reach 0.80 despite nonzero mean selection F1.
5 Conclusion
SkillCoach makes agentic skill-use measurable and trainable by separating process quality from final-task success. Its evolved rubrics diagnose failures and improve the selection of training demonstrations.
- Core contribution: SkillCoach evaluates agentic skill-use through skill selection, skill following, skill composition, and skill-grounded reflection.The framework targets reliable process behavior rather than only final outcomes.
- Core contribution: Separating trajectory-level process quality from verifier success exposes whether agents used skills reliably or passed tasks through trial and error.This distinction supports both diagnosis and better training-data selection.
- Training impact: Evolved rubrics select higher-quality demonstrations for supervised training than outcome-only filtering.The resulting supervision is grounded in observable skill-use behavior.
6 Limitations
SkillCoach’s evidence is constrained by benchmark scale and an offline training setup, leaving production-scale and deployment-time behavior untested.
- Benchmark scope: The experiments use selected skill-dependent benchmark tasks, whose scale is smaller than continuously growing production skill repositories.The benchmark still spans diverse enterprise-style domains and splits tasks by family to test unseen-workflow generalization.
- Training scope: The training study evaluates offline supervised fine-tuning with rubric-filtered trajectories, not on-policy reinforcement learning or long-term deployment feedback.
A Task Inventory
SkillCoach evaluates generalization across a structured inventory of skill-dependent tasks, with held-out task families, human-gold evaluation rubrics, and realistic distractor skills.
- Task split: The inventory contains 18 training task families with 50 instances and 10 held-out test task families with 50 instances.Task-family splitting tests generalization to unseen skill-dependent workflows rather than memorization of task-specific trajectories.
- Evaluation references: Human-gold rubrics for test tasks cover skill selection, key-step following, skill composition, and skill-grounded reflection.Two annotators construct these references from oracle materials and real rollouts for evaluation only.
- Task packages: Each task package includes an executable environment, external verifier, oracle workflow, required gold skills, distractors, and task instances.The fixed distractor setting contains two cross-task skills and three semantically near but functionally inapplicable skills.
- Distractor scaling: An expanded distractor library varies candidate-skill count and semantic similarity to probe skill-selection robustness at scale.This setting models retrieval from large public or enterprise skill hubs.
B Process-Score Aggregation and Verifier Signal
SkillCoach aggregates applicable process dimensions into a scalar process score while preserving the external verifier as an independent outcome signal. This separation diagnoses process quality even when final task success is misleading.
- Dimension design: The four skill-use dimensions are separated because failures can be localized to selection, key-step following, composition, or reflection.A scalar process score is then used for trajectory filtering, aggregate reporting, and later optimization.
- Aggregation: The default process-score weights are λsel = 0.40, λfol = 0.30, λcomp = 0.20, and λref = 0.10.Inapplicable dimensions are removed and the remaining weights are renormalized.
- Signals: Smeta(zt) measures trajectory-level skill-use quality, while sver(zt) records the external verifier result.The verifier value ⊥ represents an unexecutable or invalid verifier result.
- Interpretation: A verifier-passing trajectory can still show poor skill use, while a failed trajectory can retain useful process evidence.SkillCoach therefore filters and diagnoses with Smeta, treating sver as a separate outcome constraint.
C Rubric-Filtered SFT Details
SkillCoach uses evolved rubrics as an offline process-quality filter for supervised fine-tuning, isolating their data-selection effect from additional optimization choices. The study trains separate model variants to test which rubric components improve skill-use supervision.
- Training setup: Offline rubric filtering tests whether self-evolved rubrics identify reusable skill-use demonstrations without adding on-policy reinforcement-learning choices.The same meta score could also serve as an RL process reward, with dimension-level feedback on skill selection and key-step following.
- Training setup: 12 SFT models cover six data variants, each trained separately for Qwen3.5-4B and Qwen3.5-9B.The variants are full_R0, full_Rbest, outcome_only, Rbest_minus_order, Rbest_minus_reflect, and Rbest_minus_step.
- Training setup: Training uses full-parameter supervised fine-tuning with LLaMA-Factory for both base-model sizes.The implementation passage also specifies five epochs, a 5.0 × 10−6 learning rate, cosine scheduling, bf16 precision, and DeepSpeed ZeRO-2.
- Training setup: Skill selection is treated as a prerequisite rather than a removable scoring term in the ablation design.The removable variants target composition order, reflection, and step-related rubric components.
D Implementation Details and Prompt Templates for Self-Evolving Rubrics
The implementation turns rubric evolution into a constrained, evidence-based pipeline: extract observable steps, score multiple process dimensions, propose localized patches, and validate them before acceptance. Prompt templates enforce visible trajectory evidence and keep verifier outcomes separate from process scoring.
- Evolution protocol: 10 trajectories form each calibration set, while a held-out validation set contains 5 trajectories when enough valid data exist.Calibration supports judging and patch proposal; validation decides whether a candidate rubric is accepted.
- Evolution protocol: R0 is initialized from observable key steps, then localized patches are proposed from calibration evidence and tested against the same held-out validation set.The accepted final rubric Rbest is selected by validation performance rather than simply taking the last generated version.
- Evolution protocol: A candidate rubric must avoid hard-gate regressions and improve the soft objective Q by more than ϵ = 0.2.Acceptance also requires at least one material validation signal rather than only increased judge confidence.
- Evolution protocol: Evolution allows at most Kmax = 6 rounds and stops early after three consecutive rejections, an unusable patch, or an illegal or empty patch.Across 28 tasks, runs used 94 total rounds, averaging 3.36 rounds per task.
- Rubric dimensions: The rubric evaluates skill selection, skill following, skill composition order, result reflection, and the external verifier as a separate outcome signal.Reflection requires an explicit visible self-check, while the verifier is excluded from the process meta score.
- Prompt templates: Key-step extraction requires concrete, observable actions grounded in task, skill, oracle, verifier, and rollout evidence.Examples include reading a file, extracting a station list, computing a threshold table, writing an artifact, and running a verification command.
- Prompt templates: Skill-following key steps must match the extracted steps exactly, and critical steps require both positive and negative trajectory evidence.The rubric specification also requires actual gold and distractor skills and distinguishes reading SKILL.md from merely mentioning a skill name.