PaperlayerView exact revision on arXiv

Source-linked AI summary

OntoRAG: Enhancing Question-Answering through Automated Ontology Derivation from Unstructured Knowledge Bases

Yash Tiwari, Owais Ahmad Lone, Mayukha Pal

arXiv:2506.00664v1cs.AIcs.LG

TL;DR

Manual ontology creation is labor-intensive for large, dynamic knowledge bases, while vector RAG and GraphRAG have limitations in global sensemaking and structural coherence. OntoRAG derives a queryable ontology from unstructured documents; in electrical-relay evaluation, it reports higher comprehensiveness and diversity win rates than the compared baselines. Its reported indexing cost and reliance on domain-specific prompts constrain scalability and generalizability.

  • Problem

    Manual ontology curation is labor-intensive for massive, dynamic knowledge bases, and vector search and GraphRAG have limitations in global sensemaking and preserving source structure.

    Source: §I · pp. 1–2

  • Method

    OntoRAG transforms unstructured PDFs into a queryable ontology through web scraping, parsing, chunking, information extraction, graph construction, and ontology creation.

    Source: §II · p. 2, §I · p. 2

  • Results

    OntoRAG O2 achieved 88% comprehensiveness and 86% diversity win rates against SS, and 65% comprehensiveness and 62% diversity win rates against GraphRAG C2.

    Source: A. Qualitative Results · p. 6

  • Takeaways & Limitations

    In the electrical-relay domain, OntoRAG’s ontology-driven approach is reported to enhance global sensemaking and outperform the compared methods in comprehensiveness and diversity.

    Source: §VII · p. 6, C. Discussion · p. 6

    Indexing a 1-million-token corpus took 300 minutes on a modest virtual machine, and domain-specific prompts limit generalizability.

    Source: §VI · p. 6

Abstract

from arXiv · show

Ontologies are pivotal for structuring knowledge bases to enhance question answering (QA) systems powered by Large Language Models (LLMs). However, traditional ontology creation relies on manual efforts by domain experts, a process that is time intensive, error prone, and impractical for large, dynamic knowledge domains. This paper introduces OntoRAG, an automated pipeline designed to derive ontologies from unstructured knowledge bases, with a focus on electrical relay documents. OntoRAG integrates advanced techniques, including web scraping, PDF parsing, hybrid chunking, information extraction, knowledge graph construction, and ontology creation, to transform unstructured data into a queryable ontology. By leveraging LLMs and graph based methods, OntoRAG enhances global sensemaking capabilities, outperforming conventional Retrieval Augmented Generation (RAG) and GraphRAG approaches in comprehensiveness and diversity. Experimental results demonstrate OntoRAGs effectiveness, achieving a comprehensiveness win rate of 85% against vector RAG and 75% against GraphRAGs best configuration. This work addresses the critical challenge of automating ontology creation, advancing the vision of the semantic web.

I. INTRODUCTION

The introduction frames automated ontology creation as a response to manual curation’s limits and to vector RAG’s and GraphRAG’s challenges with global sensemaking and structural coherence. OntoRAG derives a queryable ontology from unstructured documents, aiming to preserve their relationships and hierarchy while improving comprehensiveness and diversity.

I. INTRODUCTION

  • RAG incorporates external knowledge by retrieving semantically similar documents to support LLM generation for question answering.The introduction identifies hallucination as a limitation associated with LLMs’ inability to dynamically access external knowledge.Source: §I · p. 1
  • Vector search struggles with global sensemaking and multi-hop queries because it retrieves isolated chunks rather than capturing broader relationships and context.Electrical-relay failure-mode questions may require synthesizing fragmented information across specifications, manuals, and datasheets.Source: §I · p. 1
  • GraphRAG’s clustering can disrupt source-document hierarchies and semantic dependencies, and it typically assumes a manually curated ontology.These requirements make ontology curation labor-intensive, costly, and error-prone, limiting scalability and adaptability.Source: §I · p. 1
  • Manual ontology creation is increasingly untenable for massive, dynamic knowledge bases, while LLM-based ontology learning can lack structural reasoning.Ontologies define reusable classes, properties, and relationships for data integration, sharing, and discovery.Source: §I · p. 2
  • OntoRAG derives a queryable ontology from unstructured PDFs through web scraping, parsing, chunking, information extraction, graph construction, and ontology creation.The pipeline uses community detection and LLM-based property synthesis to align ontology elements with document structure.Source: §I · p. 2
  • OntoRAG’s evaluation in the electrical-relay domain is presented as evidence of superiority over traditional and Graph RAG approaches.The paper’s stated contributions include automated ontology creation and PDF parsing, hybrid and semantic chunking, community detection, and LLM-based property synthesis.Source: §I · p. 2

II. METHODOLOGY

The methodology describes a pipeline for turning unstructured PDFs into a queryable ontology, beginning with document acquisition and parsing. It also specifies PDF-layout challenges and a table-extraction approach using coordinate information and added padding.

II. METHODOLOGY

  • OntoRAG transforms unstructured PDFs into a structured, queryable ontology through six stages, from web scraping and PDF parsing to ontology creation.The stated workflow is illustrated in Fig. 2.Source: §II · p. 2, Figure 2 · p. 3

A. Web Scraping

  • Web scraping collects electrical-relay PDFs from manufacturer websites and technical repositories into a centralized repository for later processing.The process uses BeautifulSoup and Scrapy and addresses dynamic pages and web crawling.Source: A. Web Scraping · p. 2

1) Parser Requirements:

  • Electrical-relay PDFs combine mixed-column layouts, tables with equations, and embedded images, which challenge conventional rule-based parsers.A robust parser must recognize document structures such as paragraphs, tables, and charts.Source: 1) Parser Requirements · p. 2

2) Adoption of the Unstructured Library:

  • The method adopts Unstructured to identify PDF elements and layouts using OCR, object detection, and digital parsing, but its PDFMiner text extraction struggles with tabular structure.The library identifies elements including titles, headers, text, tables, and images.Source: 2) Adoption of the Unstructured Library · p. 2

3) Enhanced Table Extraction:

  • Table extraction uses Unstructured’s coordinates to isolate table regions in PyMuPDF, with padding added to include nearby titles and footnotes.The stated padding is 100 units vertically and 20 units horizontally; isolated regions are then processed using an LLM.Source: 3) Enhanced Table Extraction · pp. 2–3

1) Hybrid Chunking:

OntoRAG prepares technical documents for ontology construction by creating coherent LLM-sized chunks, extracting and mapping structured facts, and clustering key elements into a graph of candidate classes and relationships.

1) Hybrid Chunking:

  • Title-based hybrid chunking creates contextually coherent units subject to minimum-length and LLM-context-window limits.A single-element addition may exceed the upper limit; semantic chunking also merges neighboring sentences according to embedding similarity and a manually supervised threshold balancing chunk size and coherence.Source: 1) Hybrid Chunking · p. 3, 2) Semantic Chunking · p. 3

D. Information Extraction and Mapping

  • Information extraction cleans and disambiguates text, identifies entities, converts passages into atomic facts, and maps similar entities to canonical elements.The process also structures propositions as chunk-graph JSON and adds definitions and embeddings for semantic retrieval.Source: D. Information Extraction and Mapping · p. 3

E. Knowledge Graph Construction

  • Key elements are clustered into disjoint candidate classes using name and definition embedding similarity, with batches processed independently and then merged.Class-level graph edges inherit relationship types from the original relationships between local nouns.Source: E. Knowledge Graph Construction · pp. 3–4

F. Ontology Creation

Ontology creation transforms the candidate-class knowledge graph into an ontology graph by detecting communities, extracting relationships, and synthesizing class properties.

F. Ontology Creation

  • Leiden community detection partitions the candidate-class graph into ontology classes, while inter-community relationships and LLM-synthesized properties form the ontology graph.The process derives final classes, relationships, and properties from the knowledge graph.Source: F. Ontology Creation · p. 4

1) Community Detection:

Community detection groups structurally and semantically related candidate classes into ontology classes; class properties are then aggregated and generalized for each community.

1) Community Detection:

  • Leiden partitions candidate classes into disjoint communities that represent ontology classes grouped by structural and semantic relationships.A human-in-the-loop approach guides selection of the community count to ensure meaningful classes for the electrical relay domain.Source: 1) Community Detection · p. 4

2) Class Property Extraction:

  • For each community, candidate-class properties and intra-community relationships are aggregated, then Gemini 2.5 Flash synthesizes generalized properties to handle synonyms and variations.The resulting properties are intended to be consistent and applicable to the entire class.Source: 2) Class Property Extraction · p. 4

3) Relationship Extraction:

OntoRAG projects relationships between communities and detects sub-communities to form a hierarchy; retrieval uses ontology classes at configurable levels to find source chunks and generate answers.

3) Relationship Extraction:

  • Edges between different communities become class-level relationships, while intra-community relationships support property extraction and identically labeled reciprocal edges are merged into undirected relationships.The projection excludes intra-community edges from the inter-community relationship set.Source: 3) Relationship Extraction · p. 4

4) Sub-Community Detection:

  • Sub-community detection partitions each community into finer units, with set inclusion defining a “HAS-A” hierarchy and properties and relationships extracted similarly.The resulting sub-communities increase ontology granularity.Source: 4) Sub-Community Detection · p. 4

III. RETRIEVAL METHODOLOGY

  • Retrieval matches query key elements against ontology classes at a configurable level, retrieves source chunks for top matches, expands their context windows, and supplies the combined context to Gemini 2.5 Flash.Higher ontology levels provide finer-grained classes, while lower levels provide more abstraction.Source: §III · pp. 4–5

IV. EXPERIMENTAL SETUP

The evaluation compares OntoRAG, GraphRAG, and semantic-search RAG on high-level sensemaking questions from an electrical-relay document corpus. It tests four levels of each graph-based approach against the vector-search baseline.

IV. EXPERIMENTAL SETUP

  • The systems were compared on high-level sensemaking questions over the electrical-relay corpus.The evaluation focused on global sensemaking tasks.Source: §IV · p. 5

A. Dataset

  • The evaluation used a proprietary corpus of ABB relay manuals, datasheets, and application notes, totaling about 1 million tokens in 1,600 chunks.Chunks contained 600 tokens with 100-token overlaps.Source: A. Dataset · p. 5

B. Conditions

  • Nine conditions comprised OntoRAG at four ontology levels, GraphRAG at four community levels, and semantic-search RAG as the baseline.The OntoRAG levels ranged from root to low-level granularity; GraphRAG levels ranged from root to low-level communities.Source: B. Conditions · p. 5

C. Question Generation

The study generated expert-validated questions from user personas and tasks, then evaluated answers using pairwise judgments and claim-based measures. The measures cover answer coverage, perspective variety, decision support, directness, claim counts, and claim clusters.

C. Question Generation

  • Gemini 2.5 Flash generated 125 sensemaking questions from five user personas and their tasks, with domain experts validating relevance.The personas included users such as relay technicians, and tasks included troubleshooting.Source: C. Question Generation · p. 5

D. Evaluation Metrics

  • Pairwise answer comparisons used Gemini 2.5 Flash as judge, with five replicates per question to account for LLM stochasticity.The qualitative metrics were comprehensiveness, diversity, empowerment, and directness.Source: D. Evaluation Metrics · pp. 5–6
  • Claim-based measures counted unique claims per answer for comprehensiveness and claim clusters per answer for diversity.Claim extraction used Claimify; clusters were computed with agglomerative clustering and ROUGE-L distance.Source: D. Evaluation Metrics · p. 6

E. Configuration

OntoRAG outperformed GraphRAG in comprehensiveness and diversity across levels, with O2 achieving the highest scores. The evaluation used multiple software tools and a virtual machine for indexing.

E. Configuration

  • The pipeline used Unstructured, PyMuPDF, Gemini 2.5 Flash, and Leiden community detection; indexing took about 300 minutes on a 16GB-RAM virtual machine.The machine used an Intel Xeon Platinum 8171M CPU at 2.60GHz.Source: E. Configuration · p. 6

V. RESULTS AND DISCUSSION

  • The evaluation compared ontology levels O0–O3 against GraphRAG levels C0–C3 and semantic-search RAG using qualitative and claim-based metrics.Source: §V · p. 6

A. Qualitative Results

  • OntoRAG consistently outperformed GraphRAG in comprehensiveness and diversity, with O2 achieving the highest scores.The reported trend compares OntoRAG and GraphRAG across their respective levels.Source: A. Qualitative Results · p. 6

B. Claim-Based Results

O2 slightly exceeded GraphRAG C2 on claim counts and clusters, while OntoRAG’s broader evaluation found gains in global sensemaking alongside limitations in runtime and prompt generalizability. The conclusion calls for efficiency improvements, larger-corpus evaluation, and adaptation to diverse domains.

B. Claim-Based Results

  • 35.2 claims and 14.1 clusters per answer for O2 slightly exceeded C2’s 34.8 claims and 13.8 clusters; SS had 26.5 claims and 8.0 clusters.The reported ranges were 34.9–35.3 claims and 13.9–14.2 clusters for other OntoRAG levels, and 34.5–34.8 claims and 13.6–13.8 clusters for GraphRAG levels.Source: B. Claim-Based Results · p. 6, Table II · p. 7

C. Discussion

  • The discussion reports comparable empowerment against C2 and SS’s directness advantage, while suggesting local retrieval strategies may benefit specific queries.O2’s directness win rate against C2 was 58%; SS’s directness win rate was 92%.Source: C. Discussion · p. 6

VI. LIMITATIONS

  • Indexing a 1-million-token corpus took 300 minutes, and domain-specific prompts require manual adaptation for new use cases.The discussion also notes that multiple similarity searches across ontology levels add computational overhead.Source: §VI · p. 6

VII. CONCLUSION

  • OntoRAG’s conclusion reports comprehensiveness win rates of 88% against SS and 65% against C2, while identifying computational cost and domain-specific prompts as production-readiness constraints.Future work targets efficiency, larger corpora, and diverse domains through adaptive prompting.Source: §VII · p. 6

Related papers

Loading 2506.00664v1…