Source-linked AI summary
SkillEvo: Self-Renewing Evolution Gradients from Multi-Turn Interaction Feedback
Qianxi Yan, Chunrong Chen, Jiuzhou Zhao, Min Zhang, Yongzhou Xu, Xiaochuan Xu
TL;DR
Skill evolution lacks a closed loop that converts multi-turn interaction failures into reusable feedback, while scalar governance cannot diagnose or repair structural degradation. SkillEvo combines multi-turn feedback generation with independent governance, improving over self-reflection-based evolution by 23.0 points and single-turn-QA-driven evolution by 15.4 points across its reported evaluation.
Problem
Existing skill evolution relies on manual maintenance or single-turn feedback, while scalar gates cannot localize or repair structural degradation.
Method
SkillEvo uses multi-turn user simulation to generate attributable evolution feedback and an independent governance layer to repair factual and structural degradation.
Results
23.0 points over self-reflection-based evolution and 15.4 points over single-turn-QA-driven evolution were reported across six cloud-service categories, 9 production Skills, and 98 skill-reference files.
Takeaways & Limitations
The reported results and ablation identify multi-turn interaction feedback as the source of SkillEvo’s lead, while governance limits degradation across rounds.
Takeaways & Limitations
Deployment requires human confirmation before any revised Skill reaches production.
Abstract
from arXiv · showhide
Agent Skills are today either hand-authored or produced in a single LLM generation pass, and consequently possess no closed loop through which they might improve from the interaction failures they actually cause. Recent work does close this loop, but derives its feedback from single-turn question-answering evaluation. The consequence is a sharp asymmetry: once the first round has patched the gaps that a single exchange can reveal, the evolution gradient decays, the defects that surface only across multiple turns remain invisible, and evolution stalls. Governance in these systems is likewise driven by an end-to-end verification score, a scalar gate that can reject a degraded candidate but can neither localize nor repair its structural cause. We argue that the binding constraint on sustained skill evolution is neither editing capability nor the number of iterations, but whether the evaluation feedback keeps supplying trustworthy evolution gradients. We introduce SkillEvo, in which trustworthy feedback generates the gradient and controllable governance constrains its direction. The first component recasts multi-turn user simulation from an evaluation endpoint into a feedback generator: follow-up questions expose defects layer by layer, so that every round of revision both consumes feedback and produces new feedback. The second replaces the passive rejection of a scalar gate with an independent governance layer that actively repairs factual degradation and structural bloat, preventing the gradient from drifting as degradation accumulates. Across six categories of cloud services, 9 production Skills, and 98 skill-reference files, SkillEvo surpasses self-reflection-based evolution by 23.0 points and single- turn-QA-driven evolution by 15.4 points.
1 Introduction
SkillEvo frames sustained skill evolution as a feedback and governance problem: single-turn evaluation exposes only early defects, while multi-turn feedback renews the evolution gradient and governance limits degradation.
- Skill maintenance remains largely manual, and failed interactions are not automatically consolidated into reusable knowledge.
- Single-turn feedback patches visible gaps, after which its evolution gradient decays and TSR saturates around round 2.
- SkillEvo recasts multi-turn user simulation as a feedback generator whose follow-up questions expose progressively deeper defects.
- Its independent governance layer repairs factual degradation and structural bloat so the evolution gradient does not drift across revisions.
- 51.8 points, 23.0 points, and 15.4 points are the reported improvements over original Skills, self-reflection-based evolution, and single-turn-QA-driven evolution, respectively.
2 Related work
Related work advances simulated evaluation and skill evolution but leaves feedback and governance disconnected from targeted repair. SkillEvo closes both gaps by returning multi-turn simulation to the evolution loop and diagnosing structural degradation.
- Existing dialogue simulators expose errors beyond single-turn testing but discard trajectories after evaluation instead of returning repairable failures to the Skill.
- Agent Skills remain manually maintained, while existing self-evolution methods are structurally deficient in feedback signals and evolution governance.
- Self-reflection provides no evaluation feedback, whereas single-turn QA captures only single-turn-visible gaps and its gradient decays after the first round.
- Scalar-gated governance can reject degraded candidates but cannot localize or repair dangling references, orphan files, or factual over-generalization.
- SkillEvo uses intent coverage, orthogonal responsibility separation, and collective attribution to make multi-turn interaction a feedback generator.
- SkillEvo combines dual-anchor fact consistency with graph-structural diagnosis of knowledge bloat, reference breakage, and factual over-generalization.
3 Method
SkillEvo forms a closed loop that generates attributable multi-turn feedback, revises Skills, and independently governs factual and structural degradation. Its feedback is designed around coverage, accuracy, and attributability.
- Pipeline: SkillEvo’s pipeline runs from scenario synthesis through User Agent, Verifier, Collective Attribution, Skill Optimizer, and Skill Governor.
- Pipeline: The scenario synthesizer extracts intents, behavior facts, emotion trajectory, and a human reference solution from real handled tickets.
- Trustworthy feedback: Trustworthy feedback requires simultaneous coverage, accuracy, and attributability, corresponding to intent coverage, responsibility separation, and repairability screening.
- Trustworthy feedback: The intent state machine permits normal termination only after every intent is raised and substantively addressed, preventing premature stopping and redundant turns.
- Trustworthy feedback: Samples with incomplete key-intent coverage are excluded from the agent-side denominator and attributed to Evaluation Noise.
- Attribution: Collective attribution projects only Knowledge Gaps into revision while isolating Capability Limits and Evaluation Noise.
- Governance: Fact consistency is a hard constraint checked against production and previous-round anchors, while structural consistency softly recommends repairs.
- Governance: Structural governance targets knowledge bloat, reference breakage, and factual over-generalization in the Skill’s directed knowledge graph.
4 Experiments
SkillEvo is evaluated against no-evaluation and single-turn-QA baselines on production technical-support scenarios, using task success, exposed-intent accuracy, coverage, regression, and bloat. Results show that multi-turn feedback supplies sustained evolution gradients, while governance limits cross-round degradation and structural growth.
- Experimental setup: The evaluation covers six cloud-service categories, 9 production Skills, and 98 skill-reference files from Tencent Cloud technical-support scenarios.Every ticket was escalated to a human agent, and the dataset represents failures that existing Skills did not cover.
- Experimental setup: The comparison includes Self-Reflection, single-turn QA evaluation, and SkillEvo dynamic interaction evaluation, each run for four rounds.Self-Reflection has no evaluation feedback, whereas the other methods select the best development-set version up to each round.
- Evaluation metrics: Verifier agreement with human consensus exceeds 90%, supporting its use as an automatic basis for evolution feedback.Overall TSR counts tickets as solved when the Verifier score reaches 60 without missing a key task condition.
- Main results: Single-turn QA improves TSR from 58.9 to no more than 66.4 as its gradient decays after opening-statement gaps are patched.Its evaluation gate still intercepts degradation, so performance plateaus rather than oscillating like blind editing.
- Ablation study: Replacing multi-turn interaction with single-turn QA reduces the final result to 66.4, while removing governance lowers TSR to 78.6 (−3.2).The ablation attributes SkillEvo’s 15.4-point lead over single-turn QA to the feedback source; governance primarily prevents accumulated degradation.
- Trustworthy feedback: Coverage reaches 98.9%, simulator agreement reaches 95.3%, and exposed-intent response accuracy is 71.1%.The orthogonal evaluation separates simulator coverage and fidelity from the agent’s accuracy on intents exposed during simulation.
- Controllable governance: Governance reduces knowledge bloat to 2.8% versus 16.2% without governance, while RegR declines across the three transitions.The results indicate that capability gains arise from revising existing knowledge correctly rather than merely expanding the text.
5 Conclusion
SkillEvo frames sustained skill evolution as a problem of trustworthy feedback and controllable governance, not editing capability or iteration count. On 9 production Skills, it improves TSR by 51.8 points over original Skills and 15.4 points over single-turn-QA-driven evolution.
- SkillEvo reconstructs skill self-evolution around trustworthy feedback and controllable governance.
- 51.8 points: SkillEvo improves TSR over the original Skills on 9 production Skills.
- 15.4 points: SkillEvo improves TSR over single-turn-QA-driven evolution.
- The ablation attributes SkillEvo’s lead to multi-turn interaction feedback, while trustworthiness evaluation confirms simulator coverage and fidelity.
- Reliable evolution requires a feedback loop that exposes interaction defects and isolates simulation distortion, plus governance that maintains the Skill as a structured knowledge system.
Ethics Statement
The study uses de-identified production support tickets and addresses the risk of automatically writing erroneous knowledge back to user-facing systems. Human confirmation remains required before any revision reaches production.
- Production support tickets are de-identified by removing account, order, phone, name, resource, instance, and temporary-link identifiers.
- The signal-extraction prompt forbids extracting case-specific identifying details into a Skill.
- An erroneous merged revision could be served to real users, creating an intrinsic risk for automatic knowledge write-back.
- Governance rejects candidates deleting stable baseline facts, and human confirmation is required before production deployment.
Reproducibility Statement
SkillEvo provides reimplementation materials and states two reproduction boundaries: the production-ticket dataset cannot be released, while model families can be substituted under a Generator ≠ Evaluator requirement.
- Appendix B documents pipeline prompt constraints and Verifier rubrics for reimplementing SkillEvo.
- Appendix C provides the evolution-loop pseudocode, Appendix F lists hyperparameters, and Appendix E specifies model assignments.
- The production-support dataset cannot be released because of user-privacy and commercial-confidentiality constraints.
- The method applies beyond this ticket source when multi-turn consultation logs with human reference solutions are available.
- Model choice requires Generator ≠ Evaluator, while models from different families may be substituted without altering the framework.
A Evaluation scenarios and dataset
The evaluation uses production cloud-support tickets and constrained multi-turn simulations to test reusable Skill knowledge while preventing answer leakage and structural degradation. Human-handled outcomes provide reference solutions, and independent checks determine whether extracted knowledge is resolved, reusable, and faithfully retained.
- Table 7 defines the evaluation scenarios and dataset used for SkillEvo.
- User simulation: Simulated users disclose information progressively, advance one concrete intent per turn, and adapt emotion and cooperation to the interaction.
- Scenario synthesis: The scenario synthesizer reconstructs each task from a real ticket using the opening complaint, behavior facts, emotion trajectory, target keywords, and human reference solution.
- Intent agenda: Key intents represent consultation goals whose resolution would satisfy the user, while all key and minor intents must be raised before dialogue termination.
- Knowledge extraction: Expected solutions contain only Skill-learnable rules, paths, constraints, troubleshooting steps, and other reusable knowledge, excluding case-specific or human-only operations.
- Governance: Independent inspection checks original-to-current knowledge loss and previous-to-current error introduction, while severe losses fail the candidate and minor isolated losses may be exempt.
D.2 Failed interaction trajectory
The failed trajectory shows how a plausible but incorrect first answer can survive single-turn evaluation and mislead subsequent turns. Multi-turn follow-up exposes the contradiction against the human reference as a severe core-rule failure.
- The agent initially gives the opposite of the correct renewal rule, claiming the new quota becomes available immediately.
- The simulated user continues with suspension and pay-as-you-go questions because the first answer is plausible enough to sustain the dialogue.
- Coverage reaches 1.0 because the key intent is raised, even though the agent answers it incorrectly.
- The Verifier records a severe failure because the agent’s renewal claim directly contradicts the human reference that renewal extends validity and delays new quota availability.
D.4 Attribution output
The attribution output identifies a repeated renewal-rule knowledge gap and converts it into a targeted, evidence-based Skill revision. The editor adds the missing rules while preserving existing reference content.
- The Attributor classifies the failure as a Knowledge Gap caused by the missing renewal-extension rule in the Skill.The Skill can supply this missing knowledge.
- The learning signal states that renewal extends validity, grants fresh quota on the reset date, and switches exhausted pre-reset usage to pay-as-you-go.
- The suggested change targets references/resource-pack-deduction.md and corrects the erroneous claim that renewal immediately adds quota.
- A second failed ticket exposes the same renewal-rule gap, so collective attribution merges both failures into one learning signal.
- The Editor makes a minimal edit by appending three renewal rules without deleting existing content.
- The revision remains anchored to production baseline S0, confines additions to the signaled facts, and leaves thirteen other file sections untouched.
D.6 Interaction after the update
After the update, the agent correctly explains that renewal extends validity rather than immediately refreshing quota and addresses the user’s follow-up concern. The dialogue receives a 92 verification score.
- The revised agent states the correct renewal rule in its first turn and directly addresses the user’s central concern.
- The interaction clarifies that new quota becomes available at the reset date, while pre-reset exhaustion incurs pay-as-you-go usage without suspension.
- 92: the Verifier scores the dialogue as accurate, complete, and correct in rule.
E Implementation details
SkillEvo separates generation from evaluation by assigning the editor and evaluators to different model families. The evaluation environment restricts the service agent to skill loading and read-only retrieval, and the configuration uses deployed thresholds without a sensitivity sweep.
- The Skill Editor performs bounded edits with deepseek-v4-pro, while verification, simulation, attribution, and governance run on minimax-m3.
- Using different model families prevents a model from reviewing its own edits and avoids circular dependency.
- The service agent can load Skills and perform read-only retrieval, while write tools are deregistered.
- Threshold parameters follow the deployed evaluation pipeline, and the study does not perform a sensitivity sweep.
G System architecture and design principles
The system uses nested inner and outer evolution loops: ticket-derived signals drive iterative editing, governance checks constrain revisions, and full evaluations support cross-round selection before human-confirmed rollout.
- The pipeline retrieves and de-identifies escalated tickets, extracts signals, iteratively applies evolution and governance fixes, and performs full evaluation before selection.
- Inner loop: The inner loop performs edit, check, and fix operations for up to three iterations, persisting only successfully inspected Skills.
- Outer loop: The outer loop evaluates, attributes, edits, and reevaluates for up to four rounds, with early stopping and checkpoint validation.
- Key design principles: Feedback closure converts real tickets and simulated dialogues into attributed learning signals without hand-crafted scenarios.
- Key design principles: Bounded editing patches only evidence-supported gaps that the human agent can handle but the bot cannot.
- Key design principles: Two-tier governance follows every editing round, uses a different governor model, and checks for degradation and bloat.
- Key design principles: Persisted intents, evidence, and outcomes keep each editing trajectory traceable.