PaperlayerView exact revision on arXiv

Source-linked AI summary

Ontology Generation using Large Language Models

Anna Sofia Lippolis, Mohammad Javad Saeedizade, Robin Keskisärkkä, Sara Zuppiroli, Miguel Ceriani, Aldo Gangemi, Eva Blomqvist, Andrea Giovanni Nuzzolese

arXiv:2503.05388v1cs.AI

TL;DR

Ontology engineering is complex and largely manual, and this paper investigates using LLMs to generate ontology drafts from requirements. It evaluates two prompting techniques and finds improved competency-question modelling relative to previous approaches and novice modellers, while identifying persistent structural pitfalls.

  • Problem

    Ontology engineering is complex, cognitively costly, and mostly manual, while the extent to which LLMs can support ontology generation from predefined requirements remains a research question.

    Source: §1 · p. 2

  • Method

    The authors evaluate Memoryless CQbyCQ and Ontogenia on ontology requirements and reference modules, comparing three LLMs and using independent and incremental generation setups.

    Source: §1 · p. 2, §4.2 · p. 7

  • Results

    The proposed techniques improve the proportion of modelled competency questions and surpass previous approaches and novice ontology modellers, with Ontogenia and OpenAI o1-preview the best-performing incremental combination.

    Source: Incremental Ontology Generation Results · p. 12, §8 · p. 15

  • Takeaways & Limitations

    Expert assessment aligns closely with standard and structural metrics, but qualitative evaluation remains necessary for holistic assessment of LLM-generated ontologies.

    Source: §7 · p. 14

    The methods do not adequately prevent superfluous elements, and Memoryless CQbyCQ is unsuitable for history-dependent modelling tasks.

    Source: §9.3 · p. 20

Abstract

from arXiv · show

The ontology engineering process is complex, time-consuming, and error-prone, even for experienced ontology engineers. In this work, we investigate the potential of Large Language Models (LLMs) to provide effective OWL ontology drafts directly from ontological requirements described using user stories and competency questions. Our main contribution is the presentation and evaluation of two new prompting techniques for automated ontology development: Memoryless CQbyCQ and Ontogenia. We also emphasize the importance of three structural criteria for ontology assessment, alongside expert qualitative evaluation, highlighting the need for a multi-dimensional evaluation in order to capture the quality and usability of the generated ontologies. Our experiments, conducted on a benchmark dataset of ten ontologies with 100 distinct CQs and 29 different user stories, compare the performance of three LLMs using the two prompting techniques. The results demonstrate improvements over the current state-of-the-art in LLM-supported ontology engineering. More specifically, the model OpenAI o1-preview with Ontogenia produces ontologies of sufficient quality to meet the requirements of ontology engineers, significantly outperforming novice ontology engineers in modelling ability. However, we still note some common mistakes and variability of result quality, which is important to take into account when using LLMs for ontology authoring support. We discuss these limitations and propose directions for future research.

1 Introduction

Ontology engineering is cognitively costly and largely manual, requiring domain expertise and knowledge-engineering and modelling skills. The paper investigates whether LLMs can support ontology design and presents a framework to evaluate prompting techniques and generated ontology modules.

1 Introduction

  • Ontology engineering remains complex and mostly manual, requiring domain expertise alongside knowledge-engineering and modelling skills.The authors frame these demands as a barrier to entry even when existing methodologies provide structured guidance.Source: §1 · p. 2
  • The paper asks whether LLMs can generate ontologies meeting predefined requirements, how to evaluate them, and what strengths and weaknesses they have.It introduces a framework for assessing both ontology-generation pipelines and the generated minimal ontology modules.Source: §1 · p. 2
  • The study evaluates Memoryless CQbyCQ and Ontogenia using ontology requirements and reference minimal modules, extending previous research.It also proposes criteria for comparing different pipeline setups.Source: §1 · p. 2

2 Related Work

Prior work covers ontology engineering methodologies, LLM-based ontology generation, and prompting, but leaves open how to evaluate complete ontology-generation pipelines on realistic requirements. This study compares prompting strategies against that background.

Ontology engineering methods

  • Many ontology-engineering methodologies exist, but most remain manual, and matching modelling problems to requirements is complex and time-consuming.Examples include Methontology, NeOn, Ontology Design Pattern-based agile methodologies, and LOT.Source: Ontology engineering methods · p. 3

Ontology generation with LLMs

  • Earlier LLM work often addressed entity extraction or restricted-language axioms rather than complete ontology generation from realistic requirements.NeOn-GPT evaluated one complex prompting technique on one ontology-generation example, while prior requirement-based studies reported results comparable to novice modellers.Source: Ontology generation with LLMs · pp. 3–4
  • An existing benchmark for ontology evaluation uses entire ontologies rather than minimal modules and often lacks ontology stories.The paper identifies this as a limitation for evaluating LLM outputs at wider granularity.Source: Ontology generation with LLMs · p. 4

Prompting techniques for ontology generation

  • The study compares subtask-decomposed prompting with Chain of Thought approaches, including Metacognitive Prompting, for ontology generation.Prior prompting studies examined strategies suited to different aspects of ontology generation.Source: Prompting techniques for ontology… · p. 4

3 Preliminaries

The paper defines ontology-generation terms around competency-question coverage and minimality. These definitions distinguish whether a question is modelled from whether its ontology module contains unnecessary elements.

Preliminaries

  • An ontology is defined as a set of classes, object properties, data properties, and axioms.Source: §3 · p. 4
  • A competency question is modelled when the ontology contains the elements needed to write a validation SPARQL query, regardless of modelling quality.Source: §3 · p. 5
  • A minimal ontology module for a competency question is formed by removing elements not needed for that question.An element is superfluous when it is unused in the validation SPARQL queries.Source: §3 · p. 5
  • A modelling issue is minor when exactly one object or data property is missing and adding it would make the competency question modelled.Source: §3 · p. 5

4 Methodology

The methodology builds a benchmark of competency questions, user stories, and minimal ontology modules, then tests two prompting techniques under independent and incremental generation setups. It compares multiple LLMs and uses student solutions as a baseline.

4 Methodology

  • The study creates a benchmark dataset, introduces independent and incremental ontology-generation methods, and compares GPT-4 with OpenAI o1-preview and Llama-3.1-405B-instruct-16b.GPT-4 was selected as the primary model based on earlier comparative analyses.Source: §4 · p. 5

4.1 Dataset creation

  • The benchmark contains 10 ontologies, 100 distinct competency questions, and 29 user stories from real-world and educational projects.Included ontologies had competency questions and corresponding user stories, with questions assigned to minimal modules.Source: §4.1 · p. 6
  • The authors manually removed duplicate questions and superfluous elements, then had two authors independently cross-check and evaluate the minimal modules.The modules serve as gold-standard outputs for assessment and may support future LLM fine-tuning.Source: Minimal ontology module partitioning · p. 6
  • The dataset includes 10 data-property, 25 object-property, 62 reification, and 3 restriction competency questions.Student submissions provide a baseline for comparing LLM modelling with novice ontology engineers, with and without expert feedback.Source: Dataset composition · pp. 6–7

4.2 Prompting Techniques

  • Memoryless CQbyCQ models each question without access to prior generated ontologies, reducing input context size by about 60%, then merges the outputs.The prompts include an ontology story and guidance on ontology formalization and common pitfalls.Source: §4.2 · p. 7
  • Ontogenia carries the developing ontology forward between questions and combines Metacognitive Prompting with ontology design patterns and iterative requirement checks.Unlike Memoryless CQbyCQ, it requests labels, comments, inverse relationships, and individuals; Memoryless CQbyCQ instead includes common-pitfall guidance and reduces context size.Source: §4.2 · p. 8

4.3 Ontology Generation Methods

5 Evaluation Setup

The evaluation combines CQ coverage with ontology-structure checks and expert review. The experiments compare two generation settings and apply complementary evaluation steps to the generated ontologies.

5 Evaluation Setup

  • The experiments compare independent and incremental generation using GPT-4 with both prompting techniques, and incremental generation with three LLMs on the SemanticWebCourse dataset.Two ontology engineers cross-checked CQ judgments; broader OOPS!, structural, and expert evaluations were limited to the incremental results.Source: Experimental Setup · p. 9
  • The evaluation framework checks ontology pitfalls, the proportion of modelled CQs, superfluous elements, and expert qualitative judgments.The two settings generate either an ontology for one CQ or one covering multiple CQs associated with a story.Source: Figure 2 · p. 10
  • OOPS! results report critical issues, while other detected pitfalls are excluded because they may not constitute modelling flaws.The scanner was selected for its coverage of common modelling mistakes and best practices.Source: Experimental Setup · pp. 9–10
  • CQ coverage is assessed both with and without minor issues, while structural analysis measures rates of superfluous named classes, object properties, and data properties.Two experienced knowledge engineers independently assessed usability, completeness, accuracy, and CQ modelling adequacy, with Cohen’s kappa of 0.61.Source: Experimental Setup · p. 10, Structural Analysis · p. 10, Expert Qualitative Analysis · p. 10

6 Results

Across the incremental SemanticWebCourse evaluation, Ontogenia with OpenAI o1-preview performs best on CQ modelling, and both new prompting techniques exceed student submissions and prior state of the art. Other evaluation dimensions reveal critical pitfalls, superfluous elements, and qualitative errors that qualify the coverage results.

Standard Ontology Metrics

  • OpenAI o1-preview with Memoryless CQbyCQ produces the fewest critical OOPS! issues, while Llama-3.1-405-instruct-16b produces the most with both techniques.The most common flaw is multiple domain or range axioms, which OWL interprets conjunctively and which could cause unwanted inferences or inconsistencies.Source: Standard Ontology Metrics · p. 11, Table 1 · p. 11

Independent Ontology Generation Results

  • For independent generation with GPT-4, correctly modelled CQs are 0.91 for Memoryless CQbyCQ and 0.84 for Ontogenia, rising to 0.94 and 0.89 when minor issues are ignored.For complex CQs, the corresponding scores are 0 and 0.66; the authors describe both methods as effective for single CQs when minor issues are overlooked.Source: Independent Ontology Generation Results · p. 12

Incremental Ontology Generation Results

  • Ontogenia with OpenAI o1-preview achieves the highest correctly modelled CQ count, while both new techniques exceed student submissions and prior state of the art.The incremental comparison focuses on SemanticWebCourse, which has 15 CQs for one story, versus about two CQs per story in the other datasets.Source: §6 · p. 11, Figure 3 · p. 12, Incremental Ontology Generation Results · p. 12

Structural Analysis

  • Memoryless CQbyCQ and Ontogenia yield significant superfluous elements, while Llama-3.1-405B-instruct-bf16 reaches a rate close to 40% across both techniques and all stories.GPT-4 with CQbyCQ has the fewest superfluous elements; the table compares proportions by element type across Theatre, Music, and Hospital.Source: Incremental Ontology Generation Results · p. 13, Table 2 · p. 13

Expert Qualitative Analysis

  • Expert review finds OpenAI o1-preview, especially with Ontogenia, appears to produce fewer errors overall than GPT-4, although all generated files have incorrect domains and ranges.The qualitative analysis also identifies structural flaws in Llama outputs, including inconsistent naming, redundant classes, and overlapping domains or ranges.Source: Incremental Ontology Generation Results · p. 13, Table 3 · p. 14

7 Discussion

The prompting methods outperform previous methods and student solutions in incremental ontology generation, though structural pitfalls and superfluous elements remain. Expert assessment aligns with standard and structural metrics, but qualitative evaluation is still needed for a holistic assessment.

Overall results

  • Ontogenia with OpenAI o1-preview modelled the highest percentage of CQs in incremental generation, while both prompting methods outperformed previous methods and student solutions.Experts rated o1-generated ontologies comparable to a student with Memoryless CQbyCQ and better with Ontogenia.Source: §7 · p. 14
  • In independent generation, both prompting methods performed well on single data-property and object-property CQs but worse on reifications and restrictions.Structural analysis also found more superfluous elements and critical OOPS! pitfalls than in previous work.Source: §7 · p. 14

Multidimensional evaluation

  • Expert assessments aligned closely with standard and structural metrics, yet qualitative evaluation remained necessary for holistic assessment.Source: §7 · p. 14

Superfluous elements

  • Moderate numbers of superfluous properties or equivalent classes and properties may be less serious than unmodelled CQs or more complex errors.The authors envision identifying and manually removing these elements in an ontology-engineering co-pilot setting.Source: §7 · p. 14

8 Conclusion

The authors conclude that carefully prompted commercial LLMs can support ontology drafts that meet requirements and perform on par with non-experts. They stress that evaluation must be multidimensional and that errors remain, even as such drafts may help novice engineers begin modelling.

  • Carefully prompted commercial models such as OpenAI o1 can perform on par with non-experts, while the prompting techniques improve modelled CQs over previous approaches and novice modellers.Source: §8 · p. 15
  • Evaluating generated ontologies requires multiple criteria and elaborate manual methods, not a single criterion or simple automated metrics alone.Source: §8 · p. 15
  • LLM-generated drafts can help novice ontology engineers improve modelling quality and reduce the time and effort needed to kick-start modelling, if cost is not an issue.Source: §8 · p. 15
  • Common weaknesses include erroneous domain and range restrictions, inverse-property axioms, and superfluous or overlapping classes and properties.The authors distinguish these from major errors such as wrong axiomatisation and taxonomy mistakes.Source: §8 · p. 15

9 Appendix

The appendix illustrates how the evaluation metrics distinguish CQ coverage from unnecessary ontology elements, compares LLMs and prompting trade-offs, and outlines study limitations.

9.1 Examples of Evaluation Metrics

  • A correctly modelled CQ contains the elements needed for a SPARQL query, but may also include superfluous elements; a missing data property is a minor issue.The example treats adding the missing property as an easy correction.Source: Figure 4 · p. 19, §9.1 · p. 19

9.2 Discussion on LLMs

  • o1-preview is the most promising LLM for ontology generation, while GPT-4 may serve as its replacement; Llama fares worst in expert evaluation.Although Llama models CQs correctly at a level similar to GPT-4, it produces considerable superfluous elements and critical OOPS! pitfalls.Source: §9.2 · pp. 19–20
  • Reducing GPT-4’s input context with Memoryless CQbyCQ significantly enhances its output score, but trades quality against computational efficiency and cost.The approach may suit limited-resource settings, though output can require post-processing.Source: §9.2 · p. 20

9.3 Limitations

  • Potential data leakage arises because some selected material is online, while future LLMs could train on the published gold standard.The authors separate CQs and user stories from ontology models and password-protect the dataset to mitigate leakage risks.Source: §9.3 · p. 20
  • The study needs more evaluation criteria and generalizability testing; its mostly single-module ontologies limit reuse across systems.Memoryless CQbyCQ is unsuitable for history-dependent tasks and instead fits independent pieces that humans can integrate; further work could test new domains and manually re-engineer drafts.Source: §9.3 · p. 20
Loading 2503.05388v1…