PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillForge: Forging Domain-Specific, Self-Evolving Agent Skills in Cloud Technical Support

Xingyan Liu, Xiyue Luo, Linyu Li, Ganghong Huang, Jianfeng Liu, Honglin Qiao

arXiv:2604.08618v2cs.IRcs.AIcs.SE

TL;DR

Enterprise cloud-support agents need domain-specific skills, but existing creators lack grounding and deployed failures are not systematically converted into targeted skill improvements. SkillForge grounds initial skills in historical tickets and domain knowledge, then iteratively analyzes failures, diagnoses deficiencies, and optimizes the skill. Across five real-world scenarios, domain-contextualized creation improves initial performance and self-evolution yields consistent gains across starting points, including expert-authored skills.

  • Problem

    Generic skill creators lack domain grounding, while deployed execution failures are not systematically traced to skill deficiencies for targeted refinement.

  • Method

    SkillForge grounds initial skill synthesis in historical tickets and domain knowledge, then uses Failure Analyzer, Skill Diagnostician, and Skill Optimizer stages to iteratively revise skills.

  • Results

    Across five real-world cloud technical support scenarios, domain-contextualized skills outperform generic skills, while self-evolution delivers 9–12pp Strict CR gains across three iterations regardless of starting point.

  • Takeaways & Limitations

    Automated evolution driven by deployment feedback can improve diverse initial skills and surpass manually curated expert knowledge in the studied scenarios.

  • Takeaways & Limitations

    The framework’s text-based optimization faces a knowledge ceiling because historical ambiguity, restrictions against case-specific facts, and tacit expertise limit recovery of long-tail knowledge.

Abstract

from arXiv · show

Deploying LLM-powered agents in enterprise scenarios such as cloud technical support demands high-quality, domain-specific skills. However, existing skill creators lack domain grounding, producing skills poorly aligned with real-world task requirements. Moreover, once deployed, there is no systematic mechanism to trace execution failures back to skill deficiencies and drive targeted refinements, leaving skill quality stagnant despite accumulating operational evidence. We introduce SkillForge, a self-evolving framework that closes an end-to-end creation-evaluation-refinement loop. To produce well-aligned initial skills, a Domain-Contextualized Skill Creator grounds skill synthesis in knowledge bases and historical support tickets. To enable continuous self-optimization, a three-stage pipeline -- Failure Analyzer, Skill Diagnostician, and Skill Optimizer -- automatically diagnoses execution failures in batch, pinpoints the underlying skill deficiencies, and rewrites the skill to eliminate them. This cycle runs iteratively, allowing skills to self-improve with every round of deployment feedback. Evaluated on five real-world cloud support scenarios spanning 1,883 tickets and 3,737 tasks, experiments show that: (1) the Domain-Contextualized Skill Creator produces substantially better initial skills than the generic skill creator, as measured by consistency with expert-authored reference responses from historical tickets; and (2) the self-evolution loop progressively improves skill quality from diverse starting points (including expert-authored, domain-created, and generic skills) across successive rounds, demonstrating that automated evolution can surpass manually curated expert knowledge.

1 Introduction

SkillForge addresses two enterprise skill-maintenance gaps: generic creators lack domain grounding, and deployed failures are not systematically traced to skill deficiencies for targeted refinement.

  • Generic skill creators produce poorly aligned initial skills because they lack domain grounding.
  • Existing approaches lack a systematic mechanism to trace execution failures back to skill deficiencies and drive targeted refinements.
  • SkillForge combines a Domain-Contextualized Skill Creator with automated failure analysis and skill rewriting in an end-to-end creation–evaluation–refinement loop.
  • The framework is validated on five real-world cloud technical support scenarios.

2 The Self-Evolving Skill Framework

SkillForge creates domain-grounded skills, evaluates agent failures, diagnoses skill defects, and applies targeted revisions through a continuous, VFS-mediated evolution loop.

  • 2.1 Application Context: The application evaluates customer-facing replies against human expert solutions while human support engineers review the AI outputs before sending them.
  • 2.2 Anatomy of an Agent Skill: The constrained skill definition excludes executable scripts and uses verified system tools with schemas stored in references/tools.json.
  • 2.5 Agent Execution: The agent loads skill assets through a Virtual File System, executes tasks with the available tools, and returns generated responses.
  • 2.3 Framework Overview: The framework initializes Skill_v0 from domain data, monitors execution discrepancies as Bad Cases, and iteratively produces revised skills.
  • 2.4 Domain-Contextualized Skill Creator: The Skill Creator mines workflows, tools, and domain knowledge from historical tickets and documentation, then synthesizes a structured skill.
  • 2.6 Failure Analysis: The Failure Analyzer examines each Bad Case across knowledge, tool, clarification, and style dimensions before combining results into a deterministic verdict.
  • 2.7 Diagnosis: Batch aggregation computes category frequencies and severity distributions, while the Skill Diagnostician maps issues to skill locations and generates an Optimization Plan.
  • 2.8 Optimization and Evaluation: The Skill Optimizer applies planned edits through the VFS, and offline evaluation uses low-consistency responses to trigger the same bad-case pipeline used for deployment feedback.

3 Experiments and Results

SkillForge is evaluated on production cloud-support data using expert-aligned consistency metrics, diverse starting skills, and held-out iterative evolution. Domain contextualization improves initial skills, while repeated self-evolution improves all starting points and can exceed legacy systems, with gains eventually constrained by unresolved knowledge errors.

  • 3.1 Experiments: Evaluation uses five production cloud-support scenarios containing real anonymized tickets, sequential development splits for three evolution rounds, and a held-out fourth split.Tasks are single-turn dialogues that advance ticket resolution; bad cases are identified through automated LLM-judge filtering.
  • 3.1 Experiments: Consistency Rate compares agent replies with expert historical references using Strict and Lenient variants, and the judge achieves over 90% agreement with human consensus.Strict CR counts Consistent responses, while Lenient CR includes Partially Consistent responses.
  • 3.2 RQ1: Efficacy of Domain-Contextualized Skill Creator: +4.3pp Strict CR and +3.6pp Lenient CR are achieved by S_domain over S_generic on average across all five scenarios.The largest Strict CR gain occurs in S5 at +7.20pp, despite S_generic already including mined tool schemas.
  • 3.3 RQ2: Effectiveness of the Self-Evolution Loop: +10.99, +9.23, and +11.60 Strict CR gains follow three evolution iterations from S_manual, S_domain, and S_generic, respectively.All three starting points improve, and successive cumulative gains increase at every iteration.
  • 3.3.1 Convergence and Failure Category Analysis: Marginal gains diminish across cycles as Tool, Style, and Clarification failures decline, while Knowledge failures plateau after v1 and limit further improvement.For S_manual, per-iteration gains are +4.09/+5.55/+1.35; the plateau reflects unresolved long-tail knowledge deficiencies.
  • 3.4 Comparison with Production Legacy System: +13.76pp Strict CR over the production legacy system is achieved by the skill-equipped agent on the same held-out set.The legacy system combines predefined decision-tree workflows with manually curated expert prompts.

4 Related Work

Related work spans general-purpose tool-using agents, portable agent skills, self-improvement at multiple system levels, and LLM deployment in cloud and customer-support operations. SkillForge is situated among these efforts as a domain-specific skill-evolution approach for enterprise support.

  • Autonomous Agents: ReAct established interleaved reasoning and actions, while Toolformer and ToolLLM demonstrated scalable external API invocation by language models.These works underpin tool-augmented autonomous-agent architectures.
  • Agent Skills: Agent skills package instructions, tool schemas, and reference materials, while SkillsBench reports a 16.2pp average benefit for curated skills and negligible benefit for self-generated skills.Voyager extended automatic skill acquisition through exploration and code-based verification.
  • Self-Improvement: Self-improvement research operates at output, prompt, and agent-architecture levels through methods including Self-Refine, Reflexion, ExpeL, APE, OPRO, TextGrad, and Symbolic Learning.These approaches differ in the artifact or system component they iteratively improve.
  • Cloud and Customer Support: LLM agents are increasingly applied to cloud and IT operations, including incident root-cause analysis, cloud fault diagnosis, database anomaly diagnosis, and customer-support improvement.The cited systems illustrate deployment across failure-management and support settings.

5 Conclusion

SkillForge combines domain-grounded skill creation with automated failure-driven evolution. Across five cloud support scenarios, it improves initial skills over a generic baseline and delivers consistent gains across iterations.

  • SkillForge grounds initial skill synthesis in historical tickets and domain knowledge, then uses Failure Analyzer, Skill Diagnostician, and Skill Optimizer for iterative refinement.The framework forms an end-to-end creation–evaluation–refinement loop.
  • S_domain outperforms the generic baseline by +4.3pp Strict CR and +3.6pp Lenient CR.
  • The self-evolution loop delivers consistent Strict CR gains of 9–12pp across three iterations regardless of the starting point.Starting points include expert-authored, domain-created, and generic skills.
  • Automated evolution driven by deployment feedback can surpass manually curated expert knowledge in the scenarios studied.

A Domain-Contextualized Skill Creator Details

The Domain-Contextualized Skill Creator builds initial skills from historical support data, tool usage, and domain references. It converts mined evidence into structured, conditional, and traceable skill packages operated through a sandboxed virtual file system.

  • The Creator addresses cold start by generating Skill_v0 through Workflow Mining, Tool Mining, Knowledge Extraction, and Skill Synthesis.
  • Workflow Mining: Workflow Mining transforms ticket dialogues into structured resolution traces containing problems, workflows, lessons, and exemplar responses.Resolution paths cover clarification, information gathering, diagnosis or execution, and solution delivery.
  • Tool Mining: Tool Mining selects frequently used, non-deprecated tools and records their schemas and scenario associations in tools.json.
  • Knowledge Extraction: Knowledge Extraction combines documentation search with ticket-cited references, placing decision-critical information directly in SKILL.md and detailed material in reference files.
  • Skill Synthesis: Skill Synthesis assembles mined outputs into a fixed package whose SKILL.md includes background knowledge, case-type triage, and per-case handling procedures.
  • Synthesis Constraints: Synthesis requires procedures grounded in mined resolution paths, conditional workflows, symptom–cause–resolution links, escalation paths, and reference-file offloading beyond 500 lines or 10K characters.
  • Execution Environment: The constrained skill excludes scripts and uses verified system tools, while agents access SKILL.md and references exclusively through a sandboxed VFS with logged operations.

Four-Dimension Analysis Framework

SkillForge analyzes each bad case across style, knowledge, tool, and clarification dimensions. Deterministic aggregation converts these assessments into structured failure categories, severity, priority, and diagnostic hints.

  • Each bad case is analyzed concurrently across style, knowledge, tool, and clarification dimensions, with each dimension producing severity and issue categorization.
  • Style analysis evaluates expression quality only when semantic content is correct, while knowledge analysis detects missing, incorrect, contradictory, outdated, misapplied, or unsurfaced information.
  • Tool analysis examines missed calls, wrong selections, incorrect parameters, repeated calls, result misinterpretation, and underuse; clarification analysis evaluates information-gathering strategy.
  • Deterministic rules produce failure categories, maximum severity, and a primary category, breaking ties by knowledge > tool > clarification > style.
  • An optional LLM aggregation step adds a natural-language divergence summary and actionable diagnostic hints for skill improvement.

Batch Result Aggregation

Batch aggregation groups individual failure records by category to reveal systematic issues rather than isolated cases. It summarizes severity and issue frequencies, selects representative examples, and consolidates diagnostic hints for diagnosis.

  • Batch aggregation computes severity distributions, issue-type frequencies, representative cases, and deduplicated diagnostic hints for knowledge, tool, clarification, and style categories.Representative cases are selected by severity and diversity.
  • Category-level aggregation enables the Skill Diagnostician to identify patterns across failures instead of treating each case in isolation.

D LLM-Judge Evaluation Details

SkillForge evaluates agent responses with an LLM-Judge that separates acceptability assessment from execution-trace diagnosis, then forwards partial and inconsistent cases for failure analysis. The judge uses expert references as acceptable solution points rather than unique answers and achieves over 90% agreement with human consensus.

  • Evaluation Design: The LLM-Judge compares each actual response with a human reference to assess acceptability, while the Failure Analyzer diagnoses execution failures independently.This separation lets the Judge proxy business metrics while the Analyzer focuses on trace diagnosis.
  • Evaluation Inputs: The Judge evaluates four inputs: a global ticket summary, dialogue history, human reference response, and agent actual response.Reference responses may contain multiple messages and boilerplate, which the Judge filters before comparison.
  • Failure Handoff: Cases judged inconsistent or partial are forwarded to the Failure Analyzer as bad cases for subsequent diagnosis.Each evaluation produces structured JSON containing the verdict, reference action, actual action, and reason.
  • Validation: Over 90% agreement with human consensus supports the Judge’s reliability as a proxy for expert judgment.Two domain experts independently labeled the validation subset.
  • Verdict Criteria: A response is Consistent when its core action matches the reference or correctly resolves the customer’s problem through an alternative approach.Partial consistency covers correct direction with missing details, harmless redundancy, or a less optimal but acceptable response.
  • Verdict Criteria: Responses are Inconsistent when they omit requested information, contain factual errors, redirect customers to human support, or reveal the agent’s AI identity.Boilerplate should not affect judgment, and reasonable elaboration is acceptable for very brief references unless incorrect.
  • Skill Evolution: The Diagnostician reads aggregated failure analysis and skill files, maps patterns to specific defects, and generates prioritized modification recommendations.Recommendations identify locations, content changes, evidence needs, expected impact, and risk; the Optimizer applies additive, evidence-backed edits through the VFS.

Aggregated Failure Analysis Results

Across 108 bad cases, aggregated analysis identifies recurring failures across knowledge, tool, clarification, and style dimensions. The dominant systemic issues are robotic verbosity, unnecessary clarification, and missing domain knowledge that produces incorrect guidance.

  • Aggregation: 108 bad cases were processed across four dimensions, with each case potentially contributing issues to multiple dimensions.The dimensions are Knowledge, Tool, Clarification, and Style.
  • Systemic Findings: Three systemic issues dominate: verbose robotic responses, over-clarification of already-provided information, and missing critical domain knowledge causing incorrect guidance.These patterns emerge from the aggregated Failure Analyzer statistics.

Representative Failure Cases

Representative OSS cases show how knowledge, tool, clarification, and style deficiencies combine in concrete failures. The resulting optimization plan prioritizes knowledge gaps, then communication issues, then tool invocation improvements according to impact and risk.

  • Knowledge Missing: A signed URL case shows missing access knowledge leading to unnecessary requests for error codes and RequestIDs instead of a direct accessibility confirmation.The corresponding fixes add signed-URL knowledge, improve clarification, and trigger bucket ACL verification when a signed URL is provided.
  • Incorrect Knowledge: A mirror back-to-origin case shows incorrect format knowledge causing the agent to reject valid https:// and /* configuration and recommend harmful changes.The plan instead verifies origin response status codes before modifying configuration.
  • Missing Tool: A billing diagnosis case shows generic troubleshooting replacing resource-package data queries and missing the rule against retroactive usage offsets.Recommended changes add billing queries, the deduction rule, and tool-first clarification for billing anomalies.
  • Optimization Plan: The Diagnostician maps aggregated failures to exact skill locations and produces prioritized actions with concrete changes, evidence requirements, expected impact, and risk.The optimization plan addresses knowledge:missing and knowledge:incorrect cases before clarification, style, and tool issues.
  • Optimization Plan: The clarification and style priority targets 59 over-clarification, 78 verbose, and 25 robotic cases with minimum-information, concise, natural-language, and empathy guidance.The plan requires examples from expert references and assigns low risk.
  • Optimization Plan: The tool priority targets 34 missing_call and 21 tool_missing cases by adding trigger conditions, missing tools, and tool-result interpretation guidance.It requires knowledge search and carries medium risk because tool availability and accuracy must be verified.

Analysis

The OSS case study shows that SkillForge attributes failures across dimensions, maps evidence to editable skill locations, and orders optimization by expected impact and risk.

  • Multi-dimensional Attribution: A single root cause can cascade across knowledge, tool, clarification, and style failures, so multi-dimensional attribution identifies both primary and downstream issues.Case 1 is primarily a knowledge gap, while the aggregation identifies knowledge:missing as the primary category.
  • Evidence-to-location Mapping: Diagnostic recommendations connect failure evidence to exact files and line ranges, concrete edits, evidence needs, and expected resolved cases.This structured format enables precise execution by the Skill Optimizer.
  • Prioritized Optimization: Optimization priorities balance impact and risk: knowledge gaps come first, style and clarification follow, and tool improvements come last because of external dependencies.Knowledge is prioritized as the root cause of cascading failures, while tool changes carry medium risk.
Loading 2604.08618v2…