Source-linked AI summary
pro-team at LLMs4OL 2026 Tasks Flagship and Reuse: Retrieval-Augmented Generation and Vocabulary-Constrained Filtering for Ontology Learning
Shivam Mishra, Dhannu Ram Meena, Muneendra Ojha, Krishna Pratap Singh, Kuldeep Singh
TL;DR
Ontology learning with LLMs remains limited by hallucinated or lexically inconsistent terms and weak vocabulary enforcement. This paper combines offline retrieval-augmented prompting with deterministic vocabulary filtering, achieving strong performance across both challenge tasks, including 0.8692 Semantic Graph Similarity on Task B and 0.7416 on Task A.
Problem
LLM-based ontology learning systems can hallucinate or inconsistently express terms, while existing retrieval methods do not explicitly enforce ontology-vocabulary consistency.
Method
The framework combines offline dense retrieval-augmented prompting with context management and deterministic vocabulary-constrained filtering for closed-world ontology extension.
Results
0.8692 Overall Semantic Graph Similarity was achieved on Task B, while Task A achieved 0.7416 Semantic Match Graph Similarity.
Takeaways & Limitations
Deterministic vocabulary filtering supports strong closed-world ontology extension performance and reduces out-of-vocabulary hallucinations.
Takeaways & Limitations
Formal ablation studies were not conducted, preventing precise attribution of performance gains to retrieval, prompting, and rule-based components.
Abstract
from arXiv · showhide
Ontology learning from text remains challenging despite significant progress in Large Language Models (LLMs), which can hallucinate domain terms, produce inconsistent formats, and favor hierarchical over associative relations. In the LLMs4OL 2026 Challenge, we address both the End-to-End Flagship Task (Task A) and Ontology Extension Reuse Task (Task B) using an offline retrieval-augmented few-shot prompting pipeline. Our system employs Qwen2.5-14B-Instruct with all-MiniLM-L6-v2 for demonstration retrieval, selecting the top-5 examples for Task A and top-2 for Task B. A left-truncated context-windowing strategy preserves task instructions within long prompts. For Task B, generated triples undergo deterministic vocabulary-constrained filtering, retaining triples when at least one endpoint belongs to the sample's closed term/type vocabulary and removing duplicates of the initial ontology. The approach achieves Semantic Graph Similarity of 0.8692, Term-Typing F1 of 0.9200, and Taxonomy Discovery F1 of 0.8540 on Task B, while Task A achieves 0.7416 Semantic Graph Similarity. However, no non-taxonomic relations are extracted, highlighting limitations of closed, taxonomy-oriented relation vocabularies.
1 Introduction
Ontology learning extracts concepts and relations from text to construct machine-readable ontologies, but LLMs remain vulnerable to hallucinated vocabulary and weaker non-taxonomic relation prediction. This paper proposes an offline retrieval-augmented framework with context management and vocabulary-constrained filtering for LLMs4OL tasks.
- 1 Introduction: Formal ontologies represent domain concepts, relationships, and constraints, supporting knowledge integration, interoperability, reasoning, semantic search, recommendation, and knowledge graphs,.
- 1 Introduction: Ontology learning automatically or semi-automatically extracts concepts, relations, taxonomies, and other ontology components from text, extending rule-based, statistical, and conventional NLP methods.
- 1 Introduction: The LLMs4OL benchmark standardizes evaluation of LLM-based ontology learning across Text-to-Ontology Extraction, Term Typing, Taxonomy Discovery, and Non-Taxonomic Relation Extraction,,.
- 1 Introduction: LLMs often generate hallucinated or lexically inconsistent terms and perform better on hierarchical taxonomy prediction than on non-taxonomic relation extraction.
- 1 Introduction: The paper proposes an offline Retrieval-Augmented Generation framework combining dense retrieval, instruction-guided inference, left-truncated context management, and exact vocabulary-constrained filtering for closed-world tasks.The filtering algorithm removes generated triples with out-of-vocabulary endpoints, while the offline design targets resource-constrained environments.
2 Related Work
Ontology learning has progressed from rule-based extraction to LLM-driven reasoning, while challenges remain in expressive semantic reasoning, lexical consistency, and reproducible evaluation. Retrieval-augmented prompting with deterministic vocabulary filtering addresses these limitations for LLMs4OL Tasks A and B.
- Traditional ontology learning: Traditional ontology learning used rules, statistical learning, and conventional NLP to extract concepts, taxonomies, relations, and axioms, reducing manual effort but relying on handcrafted linguistic rules.Morphologically rich languages further expose limitations caused by immature tools, complex grammar, and the need for explicit linguistic knowledge.
- LLM-based ontology engineering: LLMs extend ontology engineering through zero-shot and few-shot reasoning across generation, alignment, evaluation, maintenance, and documentation, while reducing manual annotation.However, standardized evaluation protocols and reproducible workflows remain absent.
- Benchmark context: The LLMs4OL challenge standardized datasets and evaluated Text-to-Ontology Extraction, Term Typing, Taxonomy Discovery, and Non-Taxonomic Relation Extraction across application domains [12].These benchmarks support systematic evaluation of ontology learning systems.
- Limitations: LLMs still perform considerably better on hierarchical subclass prediction than on domain, range, and disjointness axioms, making expressive semantic reasoning harder than taxonomy extraction [9].Generative systems also introduce hallucinated or lexically inconsistent ontology terms.
- Proposed approach: The proposed framework combines dense Retrieval-Augmented Generation with exact-match vocabulary-constrained filtering to improve semantic accuracy and prediction consistency for LLMs4OL Tasks A and B.Retrieval supplies contextual guidance, while filtering restricts outputs to a predefined ontology vocabulary where applicable.
3 Methodology
The framework unifies ontology construction and extension through offline retrieval-augmented prompting, LLM inference, and deterministic post-processing. It retrieves task-specific demonstrations for Qwen2.5-14B-Instruct and applies vocabulary-aware refinement to Task B outputs.
- Framework overview: The framework combines dense semantic retrieval, Qwen2.5-14B-Instruct inference, and deterministic post-processing to construct or extend ontologies from unstructured text.Its architecture retrieves semantically similar examples, builds retrieval-augmented prompts, and refines Task B triples through vocabulary-constrained filtering before JSON output.
- Task formulation: Task A constructs a primitive ontology from raw text, whereas Task B infers only new triples needed to extend a provided base ontology.The unified pipeline therefore supports both ontology construction and ontology extension while preserving the existing structure for Task B.
- Demonstration retrieval: The system retrieves training examples with all-MiniLM-L6-v2 embeddings and cosine similarity, selecting top-5 demonstrations for Task A and top-2 for Task B.Demonstration counts are tuned separately because Task A jointly requires term extraction, typing, taxonomy, and relation extraction without a provided vocabulary.
- Prompted generation: Qwen2.5-14B-Instruct generates structured ontology outputs with task-specific prompts, greedy decoding, and token budgets of 900 for Task A and 320 for Task B.Inference uses 16-bit precision and disabled sampling for deterministic outputs; prompts are left-truncated when inputs exceed the 3,072-token limit to preserve critical instructions.
- Task B post-processing: For Task B, proximity-based term-typing heuristics supplement LLM predictions before a deterministic vocabulary-constrained filter retains triples compatible with the closed candidate vocabulary.The heuristic merges instance-of triples derived from nearby predefined terms and types with generated outputs to improve recall.
4 Experiments and Results
The framework performs strongly on vocabulary-constrained ontology reuse, especially semantic matching and term typing, but unconstrained flagship extraction shows a larger exact-versus-semantic gap. Non-taxonomic relation extraction fails completely because the relation vocabulary is restricted to six taxonomic and typing strings.
- Evaluation protocol: The official evaluation script reports task-oriented metrics and Graph Similarity, including Edge F1, Neighborhood Similarity, and Taxonomy Similarity, with Exact, Partial, and Semantic match variants.These evaluation variants are used to assess generated ontologies across both challenge tracks.
- Task B: Ontology Extension: 0.8692 Semantic Match Graph Similarity and 0.9200 Term-Typing F1 demonstrate strong Task B performance under deterministic vocabulary-constrained filtering.Filtering against the canonical vocabulary narrows Exact Match Graph Similarity to 0.8480 versus Semantic Match Graph Similarity 0.8692 and avoids out-of-vocabulary hallucinations.
- Task A: Flagship: 0.7416 Semantic Match Graph Similarity exceeds 0.5353 Exact Match Graph Similarity in unconstrained Task A, indicating semantic concepts are captured despite lexical mismatches.Task A is more complex because it extracts and structures the primitive ontology without predefined vocabulary constraints.
- Cross-task limitation: 0.0000 F1 for Non-Taxonomic Relation Extraction occurs in both Task A and Task B.The system’s relation-mapping instructions define a closed set of six strictly taxonomic and typing relation strings: is-a, instance-of, disjoint with, equivalent class, part of, and has part.
5 Conclusion
The work presents an offline RAG framework for the LLMs4OL 2026 Challenge that combines Qwen2.5-14B-Instruct, sentence-transformers/all-MiniLM-L6-v2, left-side context management, and deterministic vocabulary-constrained filtering for Task B.
- 5 Conclusion: The framework combines Qwen2.5-14B-Instruct with sentence-transformers/all-MiniLM-L6-v2 in an offline Retrieval-Augmented Generation pipeline.It was developed for the LLMs4OL 2026 Challenge.
- 5 Conclusion: Left-side context-window management and deterministic vocabulary-constrained filtering for Task B prevented out-of-vocabulary hallucinations from entering final outputs.The passage also states that this combination stabilized overall predictions.
Limitations
The framework has several limitations: component contributions were not isolated through formal ablations, post-processing filtered entity vocabularies rather than semantic predicates, and key hyperparameters were chosen empirically rather than exhaustively optimized.
- Limitations: Formal ablation studies did not isolate the individual contributions of dynamic retrieval, prompt configuration, and hybrid rule-based extraction.Consequently, the distinct impact of each component cannot be definitively quantified.
- Limitations: Post-processing focused entirely on entity vocabulary filtering rather than explicit semantic predicate filtering.
- Limitations: The 3,072-token truncation limit and top-5 versus top-2 demonstration counts were set empirically from hardware constraints and preliminary dev-set evaluations, not exhaustive grid-search optimization.
Data availability statement
The study uses a dataset provided by the organizers of the LLMs4OL 2026 challenge.
- The dataset used in the study was provided by the LLMs4OL 2026 challenge organizers.
- Data availability is tied to the organizers’ provision of the challenge dataset.
- The study’s dataset forms part of the LLMs4OL 2026 challenge materials.