Source-linked AI summary
LLMs4OL: Large Language Models for Ontology Learning
Hamed Babaei Giglou, Jennifer D'Souza, Sören Auer
TL;DR
Ontology learning traditionally depends on costly manual construction, motivating the question of whether LLMs can extract and structure ontological knowledge. LLMs4OL tests this hypothesis with zero-shot prompting across three OL tasks, nine model families, and multiple knowledge domains; the results provide a proof of concept but indicate that task-specific fine-tuning is needed for practical viability.
Problem
Ontology learning requires acquiring and representing terms, types, relations, and axioms, while traditional manual construction is costly and difficult to maintain.
Method
LLMs4OL evaluates zero-shot LLM prompting for term typing, taxonomy discovery, and non-taxonomic relation extraction across diverse ontological domains and model families.
Results
The experiments demonstrate a proof of concept for LLMs4OL, but zero-shot results indicate that task-specific fine-tuning is needed for practical viability.
Takeaways & Limitations
LLMs can be investigated as a foundation for ontology learning, with the results motivating further work on specialized models, broader domains, and hybrid methods.
Takeaways & Limitations
The evaluation covers selected knowledge domains, so broader testing is needed to assess generalizability beyond those examined.
Abstract
from arXiv · showhide
We propose the LLMs4OL approach, which utilizes Large Language Models (LLMs) for Ontology Learning (OL). LLMs have shown significant advancements in natural language processing, demonstrating their ability to capture complex language patterns in different knowledge domains. Our LLMs4OL paradigm investigates the following hypothesis: \textit{Can LLMs effectively apply their language pattern capturing capability to OL, which involves automatically extracting and structuring knowledge from natural language text?} To test this hypothesis, we conduct a comprehensive evaluation using the zero-shot prompting method. We evaluate nine different LLM model families for three main OL tasks: term typing, taxonomy discovery, and extraction of non-taxonomic relations. Additionally, the evaluations encompass diverse genres of ontological knowledge, including lexicosemantic knowledge in WordNet, geographical knowledge in GeoNames, and medical knowledge in UMLS.
1 Introduction
LLMs4OL frames ontology learning as a scalable alternative to costly, manual ontology construction and tests whether LLMs can support core OL tasks across diverse models and domains.
- Ontology learning automatically identifies terms, types, relations, and potentially axioms from text to construct ontologies.
- Traditional ontology creation can be time-consuming, costly, error-prone, and impractical when knowledge evolves or experts are unavailable.
- LLMs4OL proposes leveraging LLMs as a scalable alternative for knowledge acquisition and representation.
- The paradigm investigates term typing, type taxonomy discovery, and non-taxonomic relation extraction as core ontology-learning tasks.
- Evaluations cover WordNet, GeoNames, UMLS, and schema.org, using eight domain-independent model families plus a biomedical-specific model.
2 Related Work
Prior ontology-learning research mainly uses linguistic patterns, clustering, and bootstrapping, while related LLM research studies prompting and prompt-engineering methods for knowledge extraction and task solving.
- Ontology Learning from Text: Seed-term bootstrapping methods iteratively discover specialized types and relations from unstructured text.
- Ontology Learning from Text: Traditional ontology-learning approaches primarily use lexico-syntactic patterns and clustering for term, relation, and type discovery.
- Prompting LLMs for Knowledge: Prompting feeds an input through a template with unfilled slots, allowing an LLM to generate a final string from which the output is derived.
- Prompt Engineering: Prompt engineering designs instructions for LLMs, including standard prompts and more advanced step-by-step methods such as Chain-of-Thought.
- Prompt Engineering: LLMs4OL initially prioritizes standard prompting to establish baseline capabilities before exploring more complex prompting techniques.
3 The LLMs4OL Task Paradigm
The LLMs4OL paradigm maps ontology learning to prompted LLM tasks for term typing, taxonomy discovery, and non-taxonomic relation extraction across multiple knowledge domains.
- Paradigm: LLMs4OL represents ontologies with lexical entries, conceptual types, taxonomic hierarchies, non-taxonomic relations, and axioms.
- Evaluation Scope: The evaluated domains include WordNet, GeoNames, biomedical UMLS subontologies, and schema.org.
- Paradigm: The work empirically grounds three core ontology-learning tasks using LLMs without further training or gradient updates.
- Task A - Term Typing: Term typing prompts a lexical term with optional context and domain information, producing a type through a cloze or prefix completion.
- Task B - Taxonomy Discovery: Taxonomy discovery evaluates whether pairs of types express superclass or subclass relations, using true-or-false cloze or prefix prompts.
- Task C - Non-Taxonomic Relation Extraction: Non-taxonomic relation extraction tests heterarchical semantic relations between head and tail types across possible type pairs.
4 LLMs4OL - Three Ontology Learning Tasks Evaluations
The evaluation tests LLMs across three ontology-learning tasks and multiple knowledge domains, using zero-shot and fine-tuned settings. Results show strong variation by task, domain, model, and parameter scale, while instruction tuning substantially improves performance.
- Evaluation datasets and tasks: The benchmark covers term typing, type-taxonomy discovery, and non-taxonomic relation extraction across WordNet, GeoNames, UMLS, and schema.org.Task A uses WordNet, GeoNames, and UMLS subontologies; Task B uses GeoNames, UMLS, and schema.org; Task C uses UMLS.
- Evaluation models: Eleven models span encoder-only, decoder-only, encoder-decoder, and biomedical-specific architectures, enabling comparisons across model families and parameter sizes.The evaluated models include BERT, BART, Flan-T5, BLOOM, GPT, LLaMA, and PubMedBERT.
- Zero-shot evaluations: 91.7% MAP@1 is the best term-typing result on WordNet, while GPT-4 reaches 39.4% on GeoNames and biomedical results range from 16.1% to 37.7%.The reported ordering is WordNet by GPT-3.5, GeoNames by GPT-4, SNOMEDCT US and MEDCIN by BLOOM-3b, and NCI by GPT-4.
- Fine-tuned evaluations: Instruction tuning improves Flan-T5 performance by averages of 25% for term typing, 18% for taxonomy discovery, and 3% for relation extraction.The largest Task A improvement is 45% on WordNet, while the smallest is 9% on SNOMEDCT US; the authors also report gains with increasing parameters for GPT, Flan-T5, and BLOOM.
5 Conclusions and Future Directions
The paper presents LLMs4OL as a proof of concept for using LLMs in ontology learning, supported by experiments across three OL tasks. It also identifies broader directions for improving models, coverage, evaluation, and human involvement.
- Conclusions: 11 LLMs were evaluated across three ontology-learning tasks, demonstrating the paradigm’s proof of concept.The released codebase supports replication and extension to additional models.
- Future Directions: Future work should enhance LLMs specifically for ontology learning through new architectures and fine-tuning.
- Future Directions: Broader evaluations should cover additional knowledge domains to assess how well LLM-based ontology learning generalizes.
- Future Directions: Hybrid systems combining LLMs with lexico-syntactic pattern mining and clustering may improve ontology construction.
- Future Directions: Future research should standardize metrics, build benchmarks, study ontology evolution, extract more semantic relations, and include experts in the loop.
A Apendix
The appendix documents zero-shot prompt templates for term-typing evaluation across WordNet, GeoNames, and UMLS datasets.
- WordNet: WordNet zero-shot templates are provided for Task A term typing.
- GeoNames: GeoNames zero-shot templates are provided for Task A term typing, including a place-in-country formulation.
- UMLS: UMLS zero-shot templates are provided for Task A term typing across the NCI, MEDCIN, and SNOMEDCT US sources.
A.2 Task B
Task B uses zero-shot prompt templates to discover type taxonomic relations across GeoNames, UMLS, and Schema.Org, with WordNet template material also included in the appendix.
- Task B: Task B templates cover GeoNames, UMLS, and Schema.Org for type taxonomy discovery.
- Prompt Templates: WordNet Task A templates define lexical entries L and sentences S as prompt components.
- Prompt Templates: GeoNames Task A templates use lexical entries L and sentences S for zero-shot testing.
- Prompt Templates: UMLS Task A templates provide zero-shot prompts for lexical term typing.
- Task B: Task B templates represent type pairs (a, b) or (b, a), where a is the parent and b is the child.
B Flan-T5 Training Setups and Hyperparameters
The Flan-T5 training setup fine-tunes models across OL tasks and evaluates them with zero-shot testing using multiple ontology sources. Training uses a shared AdamW-based strategy with dataset- and model-specific batch sizes and epoch counts.
- Training Setup: Flan-T5 models were fine-tuned on three tasks and evaluated using zero-shot testing across WordNet, GeoNames, UMLS, and Schema.Org.Task coverage differed by source: WordNet supported Task A, GeoNames Tasks A and B, UMLS Tasks A, B, and C, and Schema.Org Task B.
- Training Setup: Task A was treated as an 8-shot training instance, while Task B and C samples were included only when they appeared in the Task A 8-shot instances.
- Hyperparameters: All models used AdamW with a learning rate of 1e-5, while batch size varied between Flan-T5-Large and Flan-T5-XL.The batch sizes were 8 and 4, respectively.
- Hyperparameters: WordNet and Schema.Org were fine-tuned for 5 epochs, whereas UMLS was fine-tuned for 10 epochs.
C Detailed Results
This section reports detailed prompt-template results for ontology learning tasks across LLMs and datasets. Tables cover term typing and taxonomy discovery, with results presented for zero-shot testing and finetuning.
- Term typing: Tables 8 and 9 report term-typing results across prompt templates and LLMs using MAP@1 percentages.Table 8 covers WordNet and GeoName, while Table 9 covers NCI, SNOMEDCT US, and MEDCIN.
- Taxonomy discovery: Table 10 reports taxonomy-discovery results across prompt templates and LLMs using F1-score percentages.The table presents detailed results for ontology learning Task B, type taxonomy discovery.
- Evaluation setup: The reported evaluations compare zero-shot testing and finetuning across seven LLMs.An asterisk denotes finetuning model results in Tables 8–10.