PaperlayerView exact revision on arXiv

Source-linked AI summary

Frustratingly Easy Agentic RAG via Portable Skills: A Case Study in Customer Support

Istok Bako, Bonan Min, Tiffany Shao, Claire Cheng, Yashar Mehdad

arXiv:2610.14428v1cs.AI

TL;DR

Customer-support guidance requires retrieving policy-grounded steps from operational documents, while production RAG stacks can be difficult to maintain. This paper evaluates a portable skill with an LLM-generated navigation layer, reporting high guidance equivalence and evidence sufficiency on real cases.

  • Problem

    Production RAG stacks use specialized generators, multi-stage retrieval, and workflow heuristics, and are difficult to maintain as task requirements and operational content drift.

    Source: §1 · p. 1

  • Method

    A portable skill combines an operating manual, deterministic scripts, configuration, and a structured corpus, and runs in a general-purpose LLM harness.

    Source: §3 · p. 3

  • Results

    99.2% of 132 real CS cases were rated equivalent to human-approved ground truth, and 83.3% had sufficient verbatim evidence to reconstruct the procedure.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For this deployment, maintenance became editing text and corpus data rather than retraining, though corpus maintenance remains a real burden.

    Source: §9 · p. 7, §8 · p. 7

    The study covers 132 real cases on one mature, well-curated corpus and makes no cross-domain superiority claim.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Enterprise workflow automation typically requires retrieving a complete and accurate set of relevant information from large operational document corpora (SOPs, policy documents) given a specific scenario or request. In customer support (CS), for example, human specialists must determine the correct sequence of resolution steps for each case, a task that demands precise grounding in policy documentation. Meeting this bar in production typically requires complex document chunking, indexing, and retrieval pipelines, often layered with learned re-rankers, query rewriters, and bespoke agentic RAG infrastructure. We report a deployment case study of a simpler alternative: a portable skill consisting of a natural-language instruction, lightweight tools (a retrieval API, format-checking scripts), and a corpus enriched with an LLM-generated navigation layer, which is a structured "Wikipedia-style" index over the underlying documents. This agentic RAG approach is straightforward to implement (1-2 engineer days), portable across capable agentic harnesses (e.g., Claude Code), and performs strongly on the challenging task of CS resolution workflow generation, achieving 99.2% procedural equivalence to human-approved ground truth and 83.3% evidence sufficiency. It is also roughly an order of magnitude slower per case than our deployed production assistant (~90 s vs. ~8-9 s), so we claim no latency advantage. Our claim is scoped to that setting: a mature, well-curated corpus, a latency-tolerant workflow, and human review of every case. There, the skill did the work of a complex RAG stack, and maintenance became a matter of editing text and data rather than retraining models.

1 Introduction

Customer-support guidance agents must turn case information into grounded resolution steps, but production RAG stacks can be costly to build and difficult to maintain. This paper presents a portable skill and reports strong results on real cases.

1 Introduction

  • Production RAG stacks combine specialized generators, multi-stage retrieval, and workflow heuristics, and are difficult to maintain as task requirements and operational content drift.The paper motivates the skill alternative as a way to maintain the task through text and data edits rather than changes to model artifacts or pipeline components.Source: §1 · p. 1
  • A portable, model-agnostic skill packages an operating manual, deterministic scripts, configuration, and a structured corpus, with maintenance through text and corpus edits rather than retraining.A general-purpose LLM harness supplies the reasoning; the skill carries no model weights.Source: §1 · p. 1
  • 99.2% of 132 real CS cases were rated equivalent to human-approved ground truth, and 83.3% had sufficient verbatim evidence to reconstruct the procedure.The evaluation uses an LLM judge and a single hybrid retriever without production retrieval engineering.Source: §1 · p. 2
  • The paper’s contributions are a navigation layer, deployment case study, and evaluation—not a new agent algorithm.The evaluation uses procedural equivalence and evidence sufficiency to localize whether improvement belongs in the agent or corpus.Source: §1 · p. 2
  • The LLM-generated navigation layer combines intent indexing, typed links to source articles, and merged entities that consolidate procedures otherwise split across articles.Its contribution is the automatically generated combination of these properties as a swappable retrieval substrate.Source: §1 · p. 2
  • A code-enforced trust boundary rejects outputs unless every emitted snippet is a verbatim substring of its cited article, making the I/O contract auditable independently of the model.The constraint is enforced by code rather than prompting.Source: §1 · p. 2

2 Related Work

The paper situates its contribution in established RAG, agentic retrieval, evidence-quality evaluation, and LLM-as-judge work. It frames the contribution as a deployment pattern and evaluation rather than a new retrieval mechanism or agent loop.

2 Related Work

  • The paper’s contribution is a script-enforced, model-agnostic deployment unit, maintainability evidence, and an evaluation that separates procedural equivalence from evidence sufficiency—not a novel retrieval mechanism.The navigation layer is related to retrieve-over-summaries indexes, while the mechanism of an instruction-conditioned agent using tools and a retriever is described as standard.Source: §2 · p. 2

3 The Skill-Based Paradigm

The paper defines a skill as a portable, declarative package for an agentic task, executed by a general-purpose LLM harness. Its scripts enforce the output contract, while its components support portability, maintainability, auditability, and separation of concerns.

3 The Skill-Based Paradigm

  • A skill specifies an agentic task for a general-purpose LLM harness without embedding task-specific model weights, using an operating manual, deterministic scripts, configuration, and a structured corpus.The manual states the workflow, hard constraints, and output contract in editable prose.Source: §3 · pp. 2–3
  • Deterministic scripts enforce the trust boundary by checking that emitted snippets are verbatim substrings of cited source articles and rejecting non-conformant writes.The shipped validator allows only Unicode-punctuation and whitespace normalization; during benchmarking, the loader also strips withheld ground-truth fields.Source: §3 · p. 3
  • The skill is portable across capable harnesses, maintainable through prose or data edits rather than retraining, auditable independently of the model, and organized around swappable components.The stated separation of concerns includes retrieval, content access, generation, and validation.Source: §3 · p. 3

4 A Wiki-Backed Guidance Skill

The guidance skill separates retrieval over a generated wiki index from evidence quoting from authoritative source articles. It uses a swappable retrieval component and a fixed agent loop with validation.

4.1 Structured Corpus: Two Layers

  • The two-layer corpus uses about 2,700 topic-organized wiki pages for navigation and 2,881 source articles as the authoritative source for verbatim evidence.Retrieval finds wiki pages, then stable identifiers lead to source articles; the navigation layer can be re-authored independently.Source: §4.1 · p. 3
  • The LLM-generated navigation pages use intent descriptions, typed links, and merged entities to make procedures split across articles retrievable from one page.A single GPT-4.1 pass generated the layer through an update-or-create loop, without a separate clustering step.Source: §4.1 · p. 3
  • Approximately 96% of 3,168 navigation-to-source links were judged relevant and source-supported; flagged links were corrected, and all 2,881 source articles were referenced.The layer is navigational, not authoritative: emitted evidence is dereferenced from source articles and re-checked at save time.Source: §4.1 · p. 3

4.2 Retrieval Tool

  • The swappable retrieval component searches wiki pages using a hybrid of BM25 and dense similarity, with no learned re-ranker, query rewriter, or workflow-specific heuristic.A replacement retriever must map queries to ranked wiki pages and then to source-article identifiers.Source: §4.2 · pp. 3–4

4.3 Agent Loop

  • For each case, the fixed loop issues three to five hybrid queries, fetches source articles, drafts guidance, extracts up to 15 verbatim spans, and validates and saves the output.The median is roughly five queries and six fetches per case; the agent does not see prior outputs or ground truth.Source: §4.3 · p. 4

5 Evaluation

The evaluation uses 132 deidentified support cases with pre-existing, human-approved guidance withheld from the agent and used as the reference. Two LLM-judged metrics distinguish procedural equivalence from whether verbatim evidence alone can reconstruct the procedure.

5.1 Cases and Inputs

  • 132 real cases sampled from deployed traffic provide the benchmark, with personally identifiable information removed before the agent receives each case.Inputs include the contact reason, specialist notes, and user–specialist transcript.Source: §5.1 · p. 4
  • Ground-truth guidance was approved by experienced specialists before the agent existed, reused without re-audit, and withheld from the agent at inference time.Debatable ground truth is scored as failure, which the authors say biases results downward.Source: §5.1 · p. 4

5.2 Metrics

  • GUIDANCE-MATCH asks whether the candidate guidance would let a specialist resolve the case equivalently to the ground truth, allowing paraphrase and adjacent procedures.It scores procedural outcome and semantic meaning rather than surface wording.Source: §5.2 · p. 5
  • SNIPPETS-SUFFICIENT asks whether a specialist could reconstruct the ground-truth procedure using only the agent’s verbatim snippets and their source identifiers.It requires the needed procedural text to exist verbatim in a cited source article.Source: §5.2 · p. 5
  • Both metrics use majority votes across three independent judge calls; the authors estimate residual instability at roughly ±2–3 percentage points.Single-call disagreement was about 15%, higher on the harder SNIPPETS-SUFFICIENT metric (Zheng et al., 2023; Es et al., 2024; Stureborg et al., 2024).Source: §5.2 · p. 5
  • The generator and primary judge are from different vendors and model families, and the dated snapshot behind the gpt-4.1 alias cannot be recovered.The judge does not see navigation pages, preventing generated page text from leaking into its decisions.Source: §5.2 · p. 5

6 Results

On the 132-case benchmark, the skill reaches high procedural equivalence and evidence sufficiency, while a head-to-head comparison favors its navigation-layer workflow on the harder evidence metric. Ablations suggest content organization matters more than search method, though none separates statistically from the full system.

6 Results

  • 99.2% procedural equivalence and 83.3% evidence sufficiency are reported for the skill with minimal retrieval and no model training.Table 1 reports the headline rates; the 16-point gap is attributed to procedures that cannot be fully grounded in quotable evidence.Source: §6 · p. 5
  • 83.3% vs. 59.8% evidence sufficiency favors wiki search over production RAG, while procedural equivalence is near-ceiling at 99.2% vs. 96.2%.The SNIPPETS-SUFFICIENT Wilson intervals do not overlap; GUIDANCE-MATCH intervals overlap. The comparison holds corpus, cases, and evaluation protocol fixed, but not query/fetch budgets or agent behavior.Source: §6 · p. 5, Table 2 · p. 6
  • The head-to-head is an end-to-end comparison, not evidence that the retrieval representation alone explains the evidence gap.The wiki workflow can perform targeted follow-up searches after identifying missing evidence, whereas the scripted production workflow completes retrieval before composition; matched query and fetch effort is unestablished.Source: §6 · pp. 5–6, Table 2 · p. 6

6.1 Ablations

  • On SNIPPETS-SUFFICIENT, changing content organization shifts scores more than changing the search method, with merged-entity removal producing the largest drop.Hybrid scores 110/132 versus 103/132 for BM25 and 104/132 for dense search; removing merged entities scores 101/132, removing intent summaries 104/132, weakening typed links 107/132, and raw-source retrieval without the layer 103/132.Source: §6.1 · p. 6, Table 3 · p. 6
  • All ablated intervals overlap the full system’s, and paired McNemar tests distinguish no arm from it; the smallest p-value is 0.078.The authors interpret the pattern directionally: on this corpus, retrieval-surface organization appears more valuable than the search algorithm, with no single layer property accounting for the result.Source: §6.1 · p. 6, Table 3 · p. 6

7 Analysis

The analysis finds that evidence gaps mainly reflect procedural content absent from source articles, not retrieval or generation failures. It also reports an enforced verbatim-evidence contract and one illustrative guidance failure caused by an under-specified case state.

7 Analysis

  • 100% of emitted snippets satisfied the enforced verbatim-substring contract after Unicode-punctuation normalization; 99.7% were byte-identical before normalization.The 1,769 snippets came from 132 cases; the remaining 0.3% differed only by curly-versus-straight apostrophe.Source: §7 · p. 7
  • In the 22 cases failing evidence sufficiency, the dominant pattern was relevant articles and equivalent guidance but a required procedural element absent verbatim from the corpus.The missing element was typically an unwritten action such as a cross-check, fallback, tool-navigation path, or escalation target; this was qualitative author inspection, not adjudicated annotation.Source: §7 · p. 7
  • Closing these corpus-content gaps requires authoring or extending source articles, rather than improving retrieval, generation, or the skill.Appendix B provides synthetic illustrations of the gap pattern.Source: §7 · p. 7
  • Exactly one case failed GUIDANCE-MATCH because under-specified input left two procedural phases possible and the agent chose the wrong one.The authors treat this single case as illustrative, not a meaningful rate, and suggest explicit case-state enumeration as a mitigation.Source: §7 · p. 7

8 Discussion

The discussion frames maintainability as a design property of separating task competence from model weights, while acknowledging the continuing burden of navigation-layer curation. The claims are scoped to a mature, well-curated corpus and do not establish latency, cost, or robustness under load.

8 Discussion

  • Task adaptation is described as editing prose or data rather than retraining, but maintainability is a design property, not a measured result.The authors do not claim production pipelines are obsolete; learned rerankers may help more on noisier or younger corpora.Source: §8 · p. 7
  • Generated navigation pages drift as the source corpus changes, requiring structural checks, semantic audits, and removal or regeneration of flagged pages.The authors characterize this as a real text-and-data curation burden rather than an absent one.Source: §8 · p. 7
  • Ablations suggest search method is within noise while the layer’s content organization is not, but the pattern is directional and no arm differs significantly from the full system.The discussion points to content organization rather than search algorithm as the more useful focus on this corpus.Source: §8 · p. 7

9 Conclusion

The conclusion presents a declarative, model-agnostic skill for internal LLM agents and reports strong CS guidance results on real cases. Its practical lesson is to maintain the deployment by editing text and data rather than retraining.

9 Conclusion

  • 99.2% guidance equivalence and 83.3% evidence sufficiency were achieved on 132 real cases with the wiki-backed skill.The skill uses a single hybrid retriever and no production retrieval engineering.Source: §9 · p. 7
  • The skill packages an operating manual, deterministic scripts, configuration, and corpus for a general-purpose harness, without task-specific training.The authors’ deployment lesson for mature operational corpora is to maintain it by editing text and data rather than retraining.Source: §9 · p. 7
  • Search is negligible for the skill, while its wiki backend runs more navigation steps during generation than the production-RAG backend.The deployed production assistant has median per-case latency of ~8–9 s, split between retrieval (~3–4 s) and generation (~4–5 s).Source: Table 4 · p. 8

Limitations

The study’s claims are bounded by a single deployment, internal benchmark, and mature corpus; its evaluation and retrieval comparisons also leave specific questions unresolved.

Limitations

  • The benchmark covers 132 real cases on one mature, well-curated corpus, and the study makes no cross-domain superiority claim.The authors expect the recipe to transfer best to similarly mature corpora; a public procedural-support benchmark remains future work.Source: Limitations · p. 8
  • LLM judges scored quality with three-run majority voting, but human rescoring and a second judge family remain future work.Estimated residual uncertainty is ±2–3 points; the head-to-head did not hold query and fetch budgets constant, so the evidence gap at matched navigation effort is unknown.Source: Limitations · p. 8
  • At n = 132, differences between the retrieval substrate and content organization are directional rather than significant, and property claims are limited to the tested operationalizations.The study optimizes maintainability rather than latency; throughput-sensitive deployments would add batching, caching, or parallel fetch.Source: Limitations · p. 8

Ethical Considerations

The section describes privacy and human-review boundaries, distinguishes the navigation layer from prior retrieval and tool-use approaches, and presents failure inspection as qualitative.

Ethical Considerations

  • The skill runs on internal infrastructure, transmits no customer data to external systems, and assists human specialists rather than acting autonomously on accounts.Fully autonomous deployment would require safeguards beyond those discussed.Source: Ethical Considerations · p. 8

A Extended Related Work

B Failure Inspection

  • The authors qualitatively inspected 22 insufficient-evidence cases; synthetic examples illustrate missing procedure steps, but no adjudicated category counts are reported.A formal, independently labeled failure taxonomy is future work.Source: B Failure Inspection · p. 10

C Ablation Details

This appendix documents the evaluation and implementation details of the skill, including paired ablations, construction and auditing of the navigation layer, a reproducible data-free reimplementation scaffold, the output contract and validation boundary, and the frozen judges for correctness and evidence sufficiency. Ablation comparisons are directional rather than causal, and the refinement/entity-query observation is presented as a hypothesis.

C Ablation Details

  • All ablation arms used the same 132 cases and fixed skill, harness, generator, judge protocol, and k = 15, changing exactly one factor per arm.Retrieval arms kept the navigation layer unchanged; organization arms removed or weakened one layer property, and reported deltas were paired differences from the full system.Source: C Ablation Details · p. 10
  • No ablation arm differed significantly from the full system at 0.05; removing merged entities was closest (b = 15, c = 6, p = 0.078).The paired tests use two-sided exact McNemar p-values on discordant cases; the authors report deltas directionally and make no property-level causal claim.Source: C Ablation Details · p. 10, Table 5 · p. 11
  • Higher snippet coverage among cases with refinement or entity-triggered queries is reported only as an uncontrolled hypothesis, not a measured effect.The comparison was with single-query cases.Source: C Ablation Details · p. 10

D Skill Scaffold

  • The navigation layer was built fully automatically with GPT-4.1 in one pass per source article, updating or creating wiki pages and retrying failures until all 2,881 articles were covered.The loop used a fixed page schema, the running index, and each article; merged entities and typed links emerged from the update-or-create process.Source: D Skill Scaffold · p. 11
  • Approximately 96% of 3,168 navigation-to-source links passed an independent model audit, and every one of the 2,881 source articles was referenced by a navigation page.Links judged dangling, irrelevant, or insufficiently supported were removed or regenerated as maintenance.Source: D Skill Scaffold · p. 11

E Skill Code and Judge Prompts

  • The appendices provide the operating manual, tool interfaces and schema, validator, and evaluation-judge prompts without cases, article text, or organization-specific names.The data-free validator enforces the schema and verbatim-evidence constraint, and the appendix states that the materials needed to reimplement the method are included.Source: D Skill Scaffold · p. 11, E Skill Code and Judge Prompts · p. 12

E.1 README.md

  • The README describes the scaffold as sufficient to reproduce the method on any structured knowledge corpus.It contains the manual, deterministic script interfaces, output validator, and judge prompts, but no corpus or organization-specific names.Source: §E.1 · p. 12

E.2 SKILL.md: operating manual

  • The operating manual specifies a six-step case workflow that retrieves candidates, fetches source articles, drafts guidance, extracts verbatim snippets, and validates the output.The validator rejects non-conformant output; the four deterministic scripts define the tool contract and audit surface, while code enforces I/O constraints.Source: §E.2 · p. 13, §E.3 · p. 13

E.3 scripts/INTERFACES.md: tool contract and output schema

  • The output schema requires guidance and cited snippets, caps outputs at 15 snippets, and requires each snippet to be verbatim in its cited article.The contract also stamps per-case timing and is described as making outputs auditable independently of the model.Source: §E.3 · p. 13

E.4 scripts/validate_output.py: the enforced trust boundary

  • The validator checks output against the schema, rejects non-conformant writes, and enforces that each emitted snippet is a verbatim substring of its cited source article.It allows only Unicode-punctuation and whitespace normalization when checking snippet text.Source: §E.4 · pp. 13–14

E.5 Judge-prompt structure

  • Judge 1 scored standalone guidance correctness without reference guidance, while Judge 2 scored evidence sufficiency; both were frozen and majority-voted across three independent calls.Judge 2 used a calibration block because it was otherwise prone to three failure modes; the paper's numbers come from Judges 1 and 2.Source: §E.5 · p. 15

Related papers

Loading 2610.14428v1…