PaperlayerView exact revision on arXiv

Source-linked AI summary

Don't Retrieve, Navigate: Distilling Enterprise Knowledge into Navigable Agent Skills for QA and RAG

Yiqun Sun, Pengfei Wei, Lawrence B. Hsieh

arXiv:2604.14572v3cs.IRcs.AIcs.CLcs.MA

TL;DR

RAG gives LLMs external evidence but little awareness of corpus organization or unseen evidence. Corpus2Skill compiles documents into a hierarchical skill directory that agents navigate from broad summaries to documents. It improves answer quality and grounding on enterprise QA, while its benefits depend on corpus structure and entail a cost tradeoff.

  • Problem

    Traditional RAG exposes only retrieved passages, leaving the LLM without a view of corpus organization or evidence outside its retrieval window.

  • Method

    Corpus2Skill performs offline compilation into a hierarchical skill directory and lets an LLM agent navigate progressively finer summaries before retrieving documents.

  • Results

    CORPUS2SKILL achieves the highest answer-quality and retrieval-coverage scores, including Token F1 0.456 versus Agentic 0.378 and Dense 0.364, while reducing hallucination to 4.5%.

  • Takeaways & Limitations

    Corpus navigation is most useful for single-domain corpora with recoverable topical taxonomies, rather than universally replacing flat retrieval.

  • Takeaways & Limitations

    Navigation does not help on long extractive documents, homogeneous tabular corpora, or open-domain pools, and costs $0.153/query versus about 1.9× the Agentic baseline.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) grounds LLM responses in external evidence but treats the model as a passive consumer of search results, with no view of how the corpus is organized or what it has not yet seen. We present Corpus2Skill, which distills a document corpus offline into a hierarchical skill directory and lets an LLM agent navigate it at serve time, drilling from a bird's-eye view through progressively finer summaries down to documents, and backtracking when a branch is unproductive. On an enterprise customer-support benchmark, Corpus2Skill improves both answer quality and grounding over single-shot dense, hybrid, hierarchical-retrieval, and agentic RAG baselines at a moderate cost tradeoff. A ten-subset generalization study further shows that corpus navigation is not a universal replacement for retrieval: it consistently helps on single-domain corpora with a recoverable topical taxonomy, but flat retrieval remains preferable on open-domain factoid pools or homogeneous-tabular corpora that defeat top-level clustering. We characterize this scope distinction and discuss it as a design guideline for knowledge-grounded systems. Code is available at https://github.com/dukesun99/Corpus2Skill.

1 Introduction

CORPUS2SKILL reframes RAG as agentic navigation over a compiled corpus hierarchy rather than passive consumption of fixed search results. It targets queries requiring broader corpus awareness and evaluates this approach across enterprise QA and diverse corpus structures.

  • Traditional RAG gives the LLM top-k passages without exposing the rest of the corpus or whether better evidence lies outside the retrieval window.
  • Agentic RAG can issue multiple searches, but the agent still guesses productive queries without seeing the corpus structure.
  • CORPUS2SKILL compiles a corpus offline into a navigable skill tree that the agent explores at serve time, making the hierarchy the primary interface instead of an embedding index.
  • The framework recasts retrieval as agentic file browsing over a compiled skill tree and requires no embedding index or vector database at serve time.
  • A ten-subset RAGBench study finds navigation wins on most subsets, while flat retrieval remains preferable on three structurally distinct corpora.

2 Background: Skills in LLM Agents

Agent skills package reusable knowledge or workflows that agents can selectively load through progressive disclosure. CORPUS2SKILL adapts filesystem-based procedural skills into navigable informational skills describing corpus contents.

  • Agent skills package reusable procedural knowledge or multi-step workflows into modules that agents can load and follow.
  • Filesystem-based skills use a directory with SKILL.md, optional scripts, and data assets to support progressive disclosure.
  • Progressive disclosure exposes lightweight metadata first, loads full instructions only after skill selection, and keeps resources on disk until accessed.
  • CORPUS2SKILL repurposes skills to encode corpus contents: directories represent topical partitions, with SKILL.md and INDEX.md files providing increasing detail.
  • The agent follows a description → SKILL.md → INDEX.md → document workflow, navigating an information hierarchy rather than executing procedural instructions.

3 Related Work

CORPUS2SKILL combines cluster-based corpus organization with LLM-directed file navigation. Its distinguishing choices are filesystem-based progressive disclosure and standard code-execution tools instead of query-time retrieval APIs.

  • CORPUS2SKILL delegates cluster-based navigation to an LLM agent rather than requiring human selection.
  • Unlike RAPTOR, CORPUS2SKILL materializes the embed-cluster-summarize hierarchy as navigable files and replaces fixed traversal with agentic navigation.
  • GraphRAG and HiRAG organize knowledge with graph summaries but still rely on graph retrieval at query time.
  • CORPUS2SKILL builds skills from document corpora through clustering and summarization, unlike approaches that construct skills from agent trajectories.
  • The framework is distinguished by filesystem-based SKILL.md/INDEX.md progressive disclosure and standard code-execution tools rather than custom retrieval APIs.

4 CORPUS2SKILL Framework

CORPUS2SKILL has an offline compilation phase that builds a hierarchical skill forest and an online serving phase in which an agent navigates summaries before retrieving documents. Progressive disclosure, cross-references, and document lookup support targeted exploration and grounding.

  • The framework compiles corpus D into a hierarchical skill forest S offline, then serves queries through agent navigation with visibility into corpus structure.
  • At each navigation step, the agent descends through progressively finer summaries before retrieving documents through the reusable SKILL.md/INDEX.md interface.
  • Compile Phase: The compiler represents documents with summary cards combined with truncated raw text before embedding them.
  • Compile Phase: The hierarchy uses bottom-up K-Means clustering, while near-tied documents receive runner-up cross-references to support multiple navigation paths.
  • Compile Phase: The hierarchy becomes a forest of skill directories with SKILL.md roots, INDEX.md subgroups, and separately stored documents retrieved through get_document.
  • Serve Phase: Progressive disclosure preloads names and descriptions while loading file contents only on demand, balancing corpus visibility against context cost.
  • Serve Phase: The agent uses code_execution for hierarchy browsing and get_document for full text, enabling targeted backtracking and cross-branch synthesis.

5 Experiments

Experiments evaluate Corpus2Skill on WixQA, its grounding and cost, and generalization across corpus structures. Results show strong quality and grounding on coherent corpora, but navigation is not universally preferable to flat retrieval.

  • 5.2 Main Results: Corpus2Skill reaches Faithfulnessgrd 0.859 and Hallucination Rate 4.5%, retaining the answer-quality lead while substantially outperforming Agentic grounding.Agentic records 0.528 Faithfulnessgrd and 50% hallucination.
  • 5.2 Main Results: $0.153/query costs roughly 1.9× Agentic and 13–22× single-shot retrieval, although prompt caching reuses ∼70% across turns and 100% within the same top-level branch.The study also reports direct-descent and cross-branch navigation patterns.
  • 5.3 Generalization to RAGBench: Navigation loses by 0.04–0.17 F1 on open-domain factoid or homogeneous-tabular corpora when level-1 summaries become generic, repetitive, or structurally mismatched.These failures occur on HAGRID, TatQA, and CUAD.
  • 5.4 Ablation Studies: Ablations show that narrow and default trees perform similarly, larger exploration budgets mainly reduce hallucination, and Haiku lowers cost while retaining most quality.The wide tree falls to F1 0.382; budgets reduce hallucination from 6.5% to 4.5%; Haiku costs $0.093 and retains 92% of Sonnet’s F1.

6 Conclusion

CORPUS2SKILL compiles document corpora into hierarchical skill directories and replaces embedding-based retrieval with agent-driven navigation. On WixQA and across ten RAGBench subsets, it improves quality and grounding in suitable corpora, while flat retrieval remains preferable for some corpus shapes.

  • CORPUS2SKILL compiles a document corpus into a hierarchical skill directory and replaces embedding-based retrieval with agent-driven navigation.
  • On WixQA, CORPUS2SKILL improves answer quality and grounding over flat, hierarchical, and agentic baselines.
  • Across ten RAGBench subsets, navigation wins on most corpora and provides substantial gains on single-domain collections.
  • Flat retrieval remains preferable on open-domain or homogeneous-tabular corpora where top-level clustering collapses.
  • A smaller navigator retains most of the quality lead at markedly lower cost, while plain skill directories require no vector database and are easy to integrate.

Limitations

The approach has cost, routing, compilation, and corpus-scope limitations. These constraints make navigation most appropriate when query value and corpus structure justify its overhead.

  • Cost: $0.153/query makes CORPUS2SKILL about 1.9× the Agentic baseline, although Haiku 4.5 reduces cost to $0.093 while retaining most of the quality lead.
  • Cost: For high-volume, low-stakes queries, single-shot retrieval remains more cost-effective.
  • Routing errors: Top-level routing errors account for the majority of failures, especially when a query targets a document’s less obvious topic.
  • Compilation: The one-time compilation takes ∼20 minutes for WixQA’s 6,221 documents, and adding documents requires recompilation because incremental updates are unsupported.
  • Corpus scope: Navigation does not help on long extractive documents, homogeneous tabular corpora, or open-domain pools where single-shot dense retrieval already surfaces sufficient evidence.

E Full Navigation Traces

The navigation traces show agents descending from broad skills to subgroups and documents, sometimes exploring multiple branches before answering. The examples demonstrate direct, multi-document, mobile-feature, and cross-branch navigation.

  • Four traces report Factuality 1.0 and CtxR 1.0 across direct descent, multi-document retrieval, mobile quick action, and cross-branch exploration.
  • Direct descent: For changing a Wix Payments account type, the agent descended to relevant documents and answered that users must contact Wix Customer Care.
  • Multi-document retrieval: For hotel calendar synchronization, the agent navigated to hotel-booking groups and combined original Wix Hotels and HotelRunner iCal guides.
  • Mobile quick action: For the mobile Quick Action Bar, the agent found six documents and answered that deletion, rather than hiding, removes the button.
  • Cross-branch exploration: For course currency, the agent explored online-program and payment branches, combining their complementary information to locate site-level currency steps.

F Positioning Comparison

CORPUS2SKILL differs from prior systems in its input, navigation, and serving mechanisms. It converts static document corpora into informational skills and delegates branch selection to an LLM agent without dedicated runtime retrieval infrastructure.

  • Input modality: Unlike systems built from agent trajectories, CORPUS2SKILL starts from documents or text corpora.
  • Input modality: CORPUS2SKILL is the only compared system that converts a static document corpus into the skill format typically reserved for procedural knowledge.
  • Navigation mechanism: CORPUS2SKILL uniquely delegates navigation to an LLM agent that can reason about branches, backtrack, and combine evidence across branches.
  • Navigation mechanism: Prior systems use human selection, vector similarity, or graph traversal, whereas CORPUS2SKILL supports agent-driven trial-and-error navigation.
  • Serve-time infrastructure: CORPUS2SKILL eliminates dedicated query-time cluster, vector, or graph infrastructure; the runtime component is the LLM receiving pre-compiled skill files.

G Failure Analysis

The failure analysis identifies initial topic routing as the dominant bottleneck, followed by over-retrieval, synthesis errors, and failures on off-domain queries.

  • Navigation miss: 19 of 45 failures are navigation misses in which the agent retrieves no documents overlapping the gold context after choosing the wrong branch.Examples include routing PayPal setup to general payments and header-effects queries to layout groups.
  • Partial navigation: 16 queries show partial navigation: the correct general topic is found, but neighboring leaf groups contribute too many irrelevant documents.The resulting context dilution produces low-quality answers.
  • Synthesis error: 5 queries contain relevant retrieved documents but factually incorrect answers caused by over-generalization or misreading conditional instructions.These errors occur despite Context Recall≥0.6.
  • No documents retrieved: 5 off-domain queries yield zero or near-zero retrieval, and the agent declines to answer rather than hallucinating.Examples include browser-cache, Gmail password-reset, and Google Ads billing questions.
  • Takeaway: Initial topic routing remains the dominant bottleneck, while narrower top-level clusters partially mitigate failures by sharpening first-step topic separation.The narrow-tree ablation provides finer-grained top-level clustering.

H Baseline Implementation Details

The baselines compare sparse, dense, fused, hierarchical, and agentic retrieval under a shared answer-generation model and standardized evaluation setup.

  • Shared setup: All baselines use Claude Sonnet for answer generation and the same evaluation prompts.This controls the answer-generation model and prompting across methods.
  • BM25: BM25 retrieves the top-5 full articles by sparse keyword score and passes them to one LLM call.Documents are indexed at full-article level using rank_bm25.
  • Dense: Dense retrieval uses 1024-dimensional Qwen3-Embedding-0.6B vectors in a FAISS flat index, returning five cosine-similar neighbors for one LLM call.Documents and questions use separate document and query prompt templates.
  • Hybrid: Hybrid retrieval combines BM25 and dense scores with equal-weight Reciprocal Rank Fusion using k=60, then passes five passages to one LLM call.The fusion weight is α=0.5.
  • RAPTOR: RAPTOR uses UMAP, BIC-selected GMM soft clustering, recursive cluster summarization, and collapsed-tree retrieval across hierarchy levels.It shares the Dense baseline’s embedding model for consistency.
  • Agentic: The Agentic baseline lets an LLM issue BM25, dense, and hybrid searches, selecting strategies based on exact-term or conceptual-query needs.Evaluation uses the last five retrieved documents to match single-shot context windows.

I Evaluation Metrics

The evaluation combines lexical, semantic, judge-based answer-quality, grounding, coverage, hallucination, and cost measures under standardized context collection.

  • Lexical and semantic metrics: Token F1 measures the harmonic mean of token-level precision and recall, while BERTScore-F1 measures contextual-embedding similarity and is more paraphrase-robust.BERTScore-F1 uses a RoBERTa-large encoder and is averaged once per query.
  • Judge-based metrics: Factuality, Faithfulnessgrd, Context Recall, Context Precision, and Answer Relevance are LLM-judged RAGAS metrics scored on a normalized 0–1 scale.The judge uses Claude Sonnet 4.6 with greedy decoding and zero-temperature prompts.
  • Metric meanings: Faithfulnessgrd evaluates generated claims against retrieved context, Context Recall measures gold-claim coverage, Context Precision measures retrieved relevance, and Answer Relevance measures query addressing.Factuality compares the generated answer with the gold answer independently of retrieved context.
  • Hallucination Rate: Hallucination Rate is the fraction of queries with Faithfulnessgrd < 0.6, capturing how often generated content is inadequately grounded.The metric is lower-is-better and follows the RAGAS convention.
  • Context Recall standardization: All methods contribute their last five retrieved documents or passages, joined and truncated to a uniform 8,000-character context window for Context Recall judging.This standardizes comparison across single-shot, multi-turn, and navigation-based retrieval.
  • Cost metrics: Per-query cost counts all LLM input and output tokens across turns, using published pricing with Anthropic ephemeral-cache adjustments.Single-shot methods count one call, whereas Agentic and Corpus2Skill include all tool-call reasoning turns.
Loading 2604.14572v3…