PaperlayerView exact revision on arXiv

Source-linked AI summary

W-RAG: Source-Aware Retrieval for Enterprise Document Generation from Heterogeneous Knowledge Bases

Hridya Dhulipala, Rajesh Ombase, Michael Wang, Tien N. Nguyen

arXiv:2608.22081v1cs.SEcs.CLcs.IR

TL;DR

Enterprise document generation requires balanced evidence composition across heterogeneous knowledge bases, but standard RAG globally ranks retrieved candidates and can produce incomplete drafts. W-RAG combines ontology-guided retrieval, local ranking, and source-level weighting, evaluated on a new multi-domain dataset; it improves requirement satisfaction and context relevancy over existing RAG configurations. The benchmark is relatively small and synthetic, and its metrics do not fully capture every aspect of document quality.

  • Problem

    Standard RAG globally ranks evidence from heterogeneous knowledge bases, which can skew context composition and fail to preserve sources needed for complete enterprise documents.

  • Method

    W-RAG uses ontology-guided topic extraction, local ranking within each knowledge base, and source weighting to construct a balanced generation context.

  • Results

    W-RAG substantially improves Requirement Satisfaction and Context Relevancy over OG-RAG and Vanilla RAG across 16 settings.

  • Takeaways & Limitations

    Explicitly managing context distribution across knowledge bases can support more faithful and complete enterprise document generation.

  • Takeaways & Limitations

    Evaluation uses a relatively small, synthetic dataset that may not capture the diversity, scale, and noise of real-world enterprise environments.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) enables large language models to incorporate external knowledge during generation, improving factual grounding and domain adaptability. However, existing RAG pipelines assume that evidence retrieved from multiple repositories can be ranked globally using a single similarity function. While suitable for open-domain retrieval, this assumption breaks down in enterprise document generation, where heterogeneous knowledge bases (such as policies, regulations, technical documentation, and departmental guidelines) serve distinct roles and must be jointly represented in the generated document. As a result, global ranking often produces unbalanced context dominated by a subset of sources, leading to incomplete enterprise drafts. To address this limitation, we propose W-RAG, a source-aware retrieval framework that performs ontology-guided retrieval, local ranking within each knowledge base, and source-level weighting to regulate evidence composition. We further introduce a new dataset for retrieval-grounded enterprise document generation spanning multiple document types and industry domains. Experiments show that standard RAG pipelines struggle on this task, while W-RAG significantly improves document coverage and generation quality.

1 Introduction

Enterprise document generation exposes a mismatch between standard global-ranking RAG and the need to compose evidence from heterogeneous knowledge bases. W-RAG addresses this mismatch with ontology-guided retrieval, local ranking, source weighting, and a benchmark designed for multi-source enterprise writing.

  • Method: W-RAG uses ontology-guided topic modeling to retrieve passages aligned with both query wording and the target document’s thematic requirements.This first stage identifies thematically relevant evidence beyond lexical similarity.
  • Method: W-RAG ranks evidence locally within each knowledge base and applies source-level or user-defined weights to regulate each source’s contribution.The final evidence set is designed to avoid dominance by any one repository.
  • Dataset and findings: The new dataset covers retrieval-grounded enterprise document generation across multiple document types and industry domains.It is designed to require evidence composition across heterogeneous knowledge bases.
  • Dataset and findings: Standard RAG systems perform poorly despite fluent outputs, while W-RAG substantially improves enterprise document coverage and generation quality.Global-only retrieval often produces drafts that fail to cover all requested requirements.
  • Problem: Global ranking across heterogeneous knowledge bases can fail to preserve the source composition required for complete enterprise documents.The design is natural for open-domain retrieval but problematic when sources play complementary roles.

2 Related Works

Prior retrieval work extends beyond single-source retrieval through multi-hop, multigranularity, hierarchical, and ontology-based methods. However, these approaches generally target homogeneous or tightly scoped corpora, leaving enterprise synthesis across complementary knowledge bases insufficiently addressed.

  • Multi-Source and Structured Retrieval: Multi-hop and multigranularity retrieval methods combine evidence across documents but generally use homogeneous, task-specific datasets.Examples include Wikipedia and news archives.
  • Multi-Source and Structured Retrieval: Domain-specific legal and biomedical systems use hierarchical or ontology-based retrieval to reflect semantics within narrowly scoped corpora.Their focus is accuracy within one domain rather than cross-source enterprise composition.
  • Multi-Source and Structured Retrieval: Industrial evaluations report that global top-k retrieval skews evidence toward large or lexically dominant sources, overshadowing smaller essential sources.This is the enterprise-specific source-composition problem addressed by W-RAG.
  • Datasets for Knowledge-Grounded Generation: RAG benchmarks commonly target question answering, fact verification, or short-form summarization rather than long-form enterprise document generation.Multi-document reasoning datasets still typically draw from a single knowledge domain.

3 Method

W-RAG constructs a source-balanced generation context by extracting ontology-grounded topics, retrieving and ranking candidates within each knowledge base, estimating source proportions, and converting them into token budgets. It then selects locally ranked chunks under those budgets and preserves source provenance in the final context.

  • 3.1 Overview: W-RAG extracts ontology-grounded topics, retrieves candidates from each knowledge base, estimates source contributions, and builds a source-balanced context window.The total retrieval budget limits the combined number of retrieved tokens.
  • 3.2 Stage 1: Ontology-Guided Candidate Retrieval: Ontology guidance maps free-form input into structured topics that organize domain concepts and guide retrieval.For example, policy topics such as data privacy and access control can map to security and governance categories.
  • 3.2 Stage 1: Ontology-Guided Candidate Retrieval: The hybrid relevance function combines semantic similarity with ontology-based topic alignment, with λ controlling the influence of ontology guidance.Semantic similarity uses cosine similarity between dense embeddings.
  • 3.2 Stage 1: Ontology-Guided Candidate Retrieval: Chunks are ranked separately within each knowledge base, and the top-m candidates from each source are retained for later source weighting and context construction.This avoids collapsing all retrieved candidates into one global ranking.
  • 3.3 Stage 2: Source Weight Estimation: Source weights prioritize topical overlap, use category alignment as support, and treat document availability as a weak prior.The coefficients are fixed across experiments after development-stage tuning.
  • 3.4 Stage 3: Weighted Context Construction: Weights become token budgets, and W-RAG selects the highest-ranked chunks from each local candidate set until each budget is exhausted.Selected chunks are merged into a Single Context Window and annotated with their source knowledge base.

4 Dataset

The paper introduces a benchmark for requirement-driven enterprise document generation under heterogeneous knowledge-base conditions. It combines manually constructed documents, multiple enterprise genres and themes, and four functionally distinct evidence sources.

  • Dataset and Task Scope: The dataset addresses the absence of a standardized benchmark for long-form, requirement-driven enterprise document synthesis.Each sample contains an input requirement document specifying detailed constraints and objectives.
  • Dataset and Task Scope: The dataset contains 100 manually constructed requirement documents spanning policies, product launches, academic programs, and M&A due diligence.These genres represent different evidence needs, rhetorical structures, and decision-making purposes.
  • Dataset and Task Scope: The benchmark covers AI Technology & Digital Media, Healthcare & Telehealth, Climate & Sustainabilty, and Finance.These themes require integrating business, technical, legal, and market-specific information.
  • Knowledge Bases: Four heterogeneous knowledge bases provide corporate, scientific, regulatory, and market/news evidence for retrieval.Each contains approximately 500 publicly sourced documents.
  • Dataset and Task Scope: Successful generation requires combining evidence from multiple knowledge bases rather than relying on one semantically aligned source.A healthcare-AI product launch may simultaneously require corporate precedent, scientific validation, regulatory requirements, and market positioning.

5 Experiments

The experiments compare W-RAG with vanilla and ontology-guided RAG using complementary measures of document quality, retrieval relevance, and source fidelity. Across enterprise settings, source-aware retrieval improves requirement coverage and context relevance, while prescribed knowledge-base mixtures remain aligned with realized attribution.

  • Baselines and Retrieval: W-RAG uses ontology-guided topic modeling, local ranking within each knowledge base, and source-level weighting to regulate evidence composition.The baselines instead globally rank retrieved passages across knowledge bases, with OG-RAG adding ontology-guided organization before global ranking.
  • Results: W-RAG improves Requirement Satisfaction by 58.1% over OG-RAG and 109.2% over Vanilla RAG across all 16 settings.It also improves Context Relevancy by 39.1% over OG-RAG and 96.4% over Vanilla RAG.
  • Evaluation Metrics: Requirement Satisfaction measures required-content coverage, while Context Relevancy measures the proportion of retrieved passages relevant to the drafting task.Requirement checklists contain approximately 12 core requirements scored from 0 to 2, for a maximum of 24 points per document.
  • Results: In AI Technology & Digital Media product-launch documents, OG-RAG raises Context Relevancy by 64.4% over Vanilla RAG, but Requirement Satisfaction rises only 11.8%.This comparison demonstrates that retrieval relevance and document-level requirement coverage can improve by different amounts.
  • Source Fidelity: Average JS divergence scores range from 0.60 to 0.74 across domains, indicating moderate-to-strong agreement between prescribed and realized knowledge-base usage.The metric includes an additional synthesized-content dimension and assigns perfect source fidelity to a score of 1.

6 Conclusion

The conclusion presents W-RAG as a framework for enterprise documents requiring evidence from heterogeneous knowledge bases. It attributes improved requirement satisfaction and context relevance to ontology-guided topic extraction and source-aware context allocation.

  • Conclusion: W-RAG targets enterprise document generation tasks requiring evidence from multiple heterogeneous knowledge bases.The framework addresses evidence selection and source composition rather than treating all retrieved sources uniformly.
  • Conclusion: Global similarity ranking can produce imbalanced context composition in which dominant sources overshadow sources needed for complex requirements.W-RAG combines ontology-guided topic extraction with source-aware context allocation to address this composition problem.
  • Conclusion: W-RAG improves requirement satisfaction and context relevance on the benchmark dataset.The conclusion links these outcomes to explicitly managing how retrieved context is distributed across knowledge bases.

7 Limitations and Threats to Validity

The paper’s validity is bounded by a small synthetic dataset, partially subjective and incomplete evaluation metrics, limited baseline tuning, and evaluation mainly in structured enterprise settings.

  • Dataset and Task Scope: The evaluation uses a relatively small, synthetic enterprise-style dataset that may not capture real-world diversity, scale, and noise.Its scenarios are designed to reflect realistic multi-source document generation but remain narrower than operational enterprise environments.
  • Evaluation Metrics: Requirement Satisfaction and Context Relevancy involve partial subjectivity and may omit factual correctness, coherence, and downstream usability.These metrics capture selected dimensions of enterprise document quality rather than its full range.
  • Baseline Comparisons: Comparisons are limited to standard RAG baselines and exclude heavily tuned retrieval strategies, which may reduce the observed performance gap.The limitation concerns the breadth and tuning level of the comparison set.
  • Generality of the Approach: W-RAG is evaluated primarily for enterprise document generation with structured knowledge sources, leaving effectiveness in other retrieval settings unvalidated.The stated examples include open-domain question answering and less structured retrieval environments.

A.1.1 Input Requirement Documents

The dataset begins with structured requirement documents that specify the setting, business context, constraints, audience, and mandatory sections for the target enterprise draft. It covers four document types designed around enterprise writing patterns and professional conventions.

  • Input Requirement Documents: Each sample starts with an input requirement document serving as the drafting specification for the target enterprise document.Unlike short prompts or question-answer pairs, it describes organizational setting, business context, constraints, intended audience, and mandatory sections.
  • Document Types: Requirement documents are created for Policies, Product Launch documents, M&A due diligence reports, and Academic Program proposals.These four document types specify both situational context and expected structural elements of the final draft.
  • Document Design: Document design draws on common enterprise writing patterns, public institutional documentation, and professional conventions associated with each document type.The goal is to create representative requirement specifications for multi-source enterprise document generation.
  • Illustration: Figure 2 shows a sample input document for a corporate policy in the AI Technology Domain.The figure provides an example of the structured requirement-document format used in the dataset.

A.2 Ontology Construction, Topic Extraction, and Prompting

The paper builds domain–document-type ontologies and uses them to structure topic extraction and ontology-aware retrieval. New documents must be re-indexed before their topics and metadata affect retrieval.

  • Ontology Construction: The authors construct 16 ontologies, one for each domain–document-type pair, using automated drafts followed by expert review and verification.Each ontology is intended to capture concepts, categories, and semantic relations relevant to its domain and document type.
  • Ontology Construction: An ontology represents domain concepts and relationships in richer semantic categories than a taxonomy or flat classification.The policy example groups data privacy, access control, and audit logging under security, governance, and compliance categories.
  • Topic Extraction: The ontology guides topic extraction from input documents and supports comparison with topics indexed from knowledge-base documents.Extracted topics are stored as document metadata for ontology-aware retrieval.
  • Topic Extraction: Topic extraction first identifies salient phrases, then uses ontology content to align them with the ontology vocabulary and categories.The resulting topics are mapped into a structured representation.
  • Topic Extraction: Each extracted topic phrase t_i is associated with an ontology category g_i.The notation specifies the topic-to-category mapping used in the structured representation.
  • Indexing and Prompting: New knowledge-base documents influence retrieval only after re-indexing extracts topics, updates the topic index, and refreshes counts and metadata.The same process also updates source-weight estimation inputs.

A.3 LLM-as-a-Judge

The evaluation compares document quality and retrieval quality across domains and document types, showing that source-aware evidence composition matters beyond passage relevance alone. W-RAG achieves its largest benefits where documents require broader synthesis across heterogeneous knowledge bases.

  • Overall Patterns: W-RAG consistently outperforms both baselines, with larger gains for document types requiring broader synthesis across heterogeneous sources.The reported pattern separates source-aware composition from retrieval relevance alone.
  • Policies: 43.1% relative improvement in document quality and 26.6% in retrieval quality are reported for Policies versus OG-RAG.In AI Technology & Digital Media, Requirement Satisfaction increases from 34% to 78% relative to OG-RAG.
  • Policies: 76% versus 68% Requirement Satisfaction is achieved by W-RAG versus OG-RAG despite Context Relevancy of 72% versus 70%.The comparison indicates that marginal relevance gains do not necessarily produce equivalent document-quality gains.
  • Product Launch: 44.6% improvement in document quality and 33.9% in retrieval quality are reported for Product Launch, with Healthcare scores rising from 33% to 63% and 45% to 71%.The analysis attributes product-launch difficulty to combining technical, market, regulatory, and organizational evidence types.
  • Merger & Acquisitions: 35.3% and 41.8% under vanilla RAG rise to 70.8% and 78.8% under W-RAG for averaged Merger & Acquisitions Requirement Satisfaction and Context Relevancy.The corresponding relative gains over OG-RAG are 61.7% and 45.8%.
  • Knowledge-Base Influence: W-RAG changes realized source attribution when the prescribed knowledge-base mixture is reweighted.This result shows that the generated document content changes with the type of information drawn from heterogeneous knowledge bases.
Loading 2608.22081v1…