PaperlayerView exact revision on arXiv

Source-linked AI summary

From human experts to machines: An LLM supported approach to ontology and knowledge graph construction

Vamsi Krishna Kommineni, Birgitta König-Ries, Sheeba Samuel

arXiv:2403.08345v1cs.CL

TL;DR

Building ontologies and KGs requires substantial interdisciplinary expertise, and the paper explores whether open-source LLMs can automate parts of that work. Its pipeline generated KGs from deep-learning publications, with results suggesting potential to reduce effort while retaining human evaluation.

  • Problem

    Ontology and KG construction requires multiple stakeholders and expertise, while increasingly expressive ontologies bring design and data-quality challenges.

    Source: §1 · p. 2

  • Method

    The authors use open-source LLMs to generate competency questions, create an ontology, answer questions from scholarly publications with RAG, construct KGs, and evaluate outputs against ground truth.

    Source: §1 · p. 2, §3 · p. 5, §5 · p. 8

  • Results

    The best KGs correlated 142 of 203 KG individuals with CQ answers across five publications, while the judge and human annotator disagreed on 42 of 200 evaluated answers.

    Source: §4 · pp. 6–7

  • Takeaways & Limitations

    The study suggests LLMs can potentially reduce effort in ontology and KG creation, while presenting them as assistants to human experts.

    Source: §6 · p. 8

    Generated KGs can be inconsistent: the LLM may label individuals “Not Specified,” omit them, or duplicate values without producing meaningful KGs.

    Source: §5 · p. 8

Abstract

from arXiv · show

The conventional process of building Ontologies and Knowledge Graphs (KGs) heavily relies on human domain experts to define entities and relationship types, establish hierarchies, maintain relevance to the domain, fill the ABox (or populate with instances), and ensure data quality (including amongst others accuracy and completeness). On the other hand, Large Language Models (LLMs) have recently gained popularity for their ability to understand and generate human-like natural language, offering promising ways to automate aspects of this process. This work explores the (semi-)automatic construction of KGs facilitated by open-source LLMs. Our pipeline involves formulating competency questions (CQs), developing an ontology (TBox) based on these CQs, constructing KGs using the developed ontology, and evaluating the resultant KG with minimal to no involvement of human experts. We showcase the feasibility of our semi-automated pipeline by creating a KG on deep learning methodologies by exploiting scholarly publications. To evaluate the answers generated via Retrieval-Augmented-Generation (RAG) as well as the KG concepts automatically extracted using LLMs, we design a judge LLM, which rates the generated content based on ground truth. Our findings suggest that employing LLMs could potentially reduce the human effort involved in the construction of KGs, although a human-in-the-loop approach is recommended to evaluate automatically generated KGs.

1 Introduction

Ontology and KG construction draws on interdisciplinary expertise and involves consequential design and data-quality challenges. The paper explores LLM-supported construction, using deep-learning methods in biodiversity publications as a test case.

1 Introduction

  • Ontology and KG engineering requires multiple stakeholders, while increasing ontology expressiveness adds design complexity and challenges for accuracy, scalability, and knowledge depth.The conventional process gathers competency questions from domain experts, develops an ontology, structures unstructured data, and populates the KG.Source: §1 · p. 2
  • LLMs’ language-understanding and generation capabilities motivate their use in knowledge engineering.The paper notes their adoption across fields including information retrieval and language translation.Source: §1 · p. 2
  • The study explores using LLMs to reduce time and human effort across competency-question collection, ontology creation, and ontology population.It tests the pipeline on deep-learning methods extracted from scholarly publications in biodiversity.Source: §1 · p. 2
  • Capturing models, architectures, hyperparameters, and related details in structured form supports reproducibility, trust, and validation of deep-learning studies.The paper identifies this need as motivation for its biodiversity-domain example.Source: §1 · p. 2

2 Related work

Prior work uses LLMs for ontology and KG tasks, but the paper distinguishes its approach by reusing an existing ontology and building a comprehensive pipeline with open-source models. It also retains human validation and evaluation.

2 Related work

  • LLM research has addressed ontology creation, augmentation, completion, and learning, including KG extraction from subject-relation-object statements and concept hierarchies limited to subconcept relations.The paper contrasts its reuse of existing ontology concepts and relations with the latter approach.Source: §2 · p. 3
  • The paper reuses an existing domain ontology and involves humans in validating and evaluating LLM-generated content.It notes that LLMs may improve ontologist productivity but do not remove the need for expert collaboration and community consensus.Source: §2 · p. 3
  • The authors report that prior work largely used OpenAI API models [17] [22] [26], whose usage costs can rise quickly at scale, and present their open-source approach as a comprehensive pipeline.They describe it as the first such approach to construct ontologies and KGs using open-source LLMs, to the best of their knowledge.Source: §2 · p. 3

3 Methods

The six-component pipeline combines competency questions, ontology creation, retrieval-augmented answering, KG construction, and evaluation. It uses human-reviewed questions and ground truth while automating several subsequent steps with LLMs.

3 Methods

  • The pipeline comprises Data Collection, CQ generation, Ontology creation, CQ answering, KG construction, and Evaluation, with human involvement in its second and sixth stages.Prompts, code, evaluation results, and data are reported as publicly available.Source: §3 · p. 3, Figure 1 · p. 4
  • The study reused a prior systematic-review dataset and selected five of 61 curated publications; two domain experts reviewed ChatGPT-3.5-generated abstract-level CQs.The experts refined existing questions and added new ones.Source: §3 · p. 4
  • After comparing Llama 2-70B, Mixtral 8x7B, and Falcon-40B, the authors used Mixtral 8x7B from ontology creation onward.They report that Mixtral outputs were comparatively superior and used it zero-shot.Source: §3 · p. 4
  • Ontology creation first extracts concepts and relationships from CQs, then builds an ontology using an in-context structure and PROV-O as a foundational ontology for reuse.The extracted CQ concepts and relationships are incorporated into the ontology.Source: §3 · p. 5
  • RAG retrieves answers to all CQs from the five selected publications, after which basic text processing removes redundant or repetitive content.Processed answers, their questions, and the generated ontology are then supplied to the LLM to extract entities and map them onto the ontology for KG construction.Source: §3 · p. 5
  • A judge LLM scores CQ answers against human-generated ground truth and checks whether extracted KG concepts appear in the corresponding answers.Answer scores from 0–10 are classified as Right at 6 or above, Wrong below 3, and Partial otherwise; one expert annotated five articles.Source: §3 · p. 5

4 Results

The pipeline produced a DLProv ontology and KGs from five publications, with measured alignment between KG individuals and CQ answers. The results also report CQ coverage and disagreement between the human annotator and judge LLM.

4 Results

  • The pipeline generated 40 CQs covering DL-pipeline steps from raw data and preprocessing through models, hyperparameters, software, hardware, security, bias, and ethics.The supplied passage lists these topics as the CQs’ scope.Source: §4 · pp. 5–6
  • The generated DLProv ontology contains 45 classes, 41 relationships, and 365 axioms, reusing PROV-O with subclasses under three PROV-O classes and no reused object properties.When MLSchema was supplied for reuse, only one class and no object properties were reused.Source: §4 · p. 6
  • 42 of 200 evaluated RAG-generated CQ answers received different assessments from the human annotator and LLM judge.The answers covered the first five publications in the dataset.Source: §4 · p. 6
  • The pipeline constructed KGs describing deep-learning pipeline method information extracted from five scholarly publications.The paper shows an excerpt describing provenance of results from a DL-method publication.Source: §4 · p. 6
  • 142 of 203 KG individuals were correlated with CQ answers in the best KGs across five publications.The study generated multiple KGs using two prompts and two RAG-generated answer sets; Table 1 compares aligned-individual percentages across prompt and answer combinations.Source: §4 · p. 7, Table 1 · p. 7

5 Discussion

The discussion reports that a six-component pipeline was more effective than attempting KG construction in one step, while identifying output quality, consistency, and prompt sensitivity as continuing challenges. It also describes using an open-source judge LLM and text processing to evaluate and refine generated content.

5 Discussion

  • The initial single-step attempt produced little to no KG content, prompting the authors to divide the process into six components.The authors note known LLM problems including hallucination and prompt sensitivity.Source: §5 · p. 7
  • The judge LLM disagreed with the human evaluator on 42 out of 200 cases, often labeling human-annotated “partial” answers as wrong.Automatic text processing removed redundant and repetitive segments before evaluation.Source: §5 · p. 7
  • Generated KGs sometimes omit individuals or label them “Not Specified,” and repeated values can produce multiple individuals for one class and meaningless KGs.The authors attribute these inconsistencies to difficulty capturing intact information from the provided content.Source: §5 · p. 8
  • Minor prompt changes often altered outputs, while in-context examples helped reduce variability and hallucinated content.The authors refined prompts through trial and error after reviewing initial results.Source: §5 · p. 8
  • The authors devised an open-source judge LLM because the evaluated RAG tools functioned fully only with an OpenAI API key.The judge rates generated content against ground-truth values.Source: §5 · p. 8

6 Conclusion

The authors conclude that open-source LLMs can assist ontology and KG creation with less effort and less semantic-web expertise. They plan to improve and test the pipeline across hardware and models, while noting prompt sensitivity and anticipating hardware-related output variation.

6 Conclusion

  • The pipeline follows established ontology engineering practices and suggests LLMs can act as assistants to human experts, lowering the effort and semantic-web expertise required for ontology and KG creation.The authors frame this as potential for broader use.Source: §6 · p. 8
  • Future work will test the pipeline on different hardware and with different open-source LLMs to examine possible result variations.The authors anticipate hardware-related variation due to differences in variable initializations.Source: §6 · p. 8
  • The authors identify high prompt sensitivity and plan to explore mapping the generated ontology with other ML/DL ontologies.They note that minor prompt-design changes could significantly affect outputs.Source: §6 · p. 8

Related papers

Loading 2403.08345v1…