PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM-Assisted Discovery of Typed Semantic Links for Ontology Network Construction

Nouha Hayouni, Sheeba Samuel, Alsayed Algergawy

arXiv:2610.01393v1cs.CLcs.AI

TL;DR

Ontology networks need typed semantic links across heterogeneous domains, but current alignment, embedding, and LLM approaches leave gaps in generating them. The paper combines contextual embeddings, candidate filtering, and GPT-4o link generation; on ReproduceMeON, it reports 80.19% overall precision and F1 = 0.890, outperforming the best similarity baseline. The authors note that generalizability beyond this network and repeated-run output variance remain untested.

  • Problem

    Existing approaches have difficulty providing scalable, typed, justified semantic links across ontologies, and similarity alone does not distinguish valid from invalid high-similarity pairs.

    Source: §1 · p. 1

  • Method

    The pipeline combines domain-adapted DistilBERT embeddings, clustering-based candidate filtering, and GPT-4o generation of typed, justified links using structured prompts.

    Source: §1 · p. 2, §3.4 · p. 5

  • Results

    80.19% overall precision on 429 validated relationships and F1 = 0.890 on a balanced evaluation set, versus Sentence-BERT’s F1 = 0.581.

    Source: §1 · p. 2, §5 · p. 12

  • Takeaways & Limitations

    On the filtered candidate set, similarity methods alone achieve AUC ≈0.5, while the evaluated pipeline generates links with expert-validated precision.

    Source: §4.5 · p. 9, §4.6 · p. 11

    Evaluation is limited to ReproduceMeON, and no repeated-run experiments quantified GPT-4o output variance or prompt sensitivity.

    Source: §4.7 · p. 12

Abstract

from arXiv · show

Constructing typed, justified semantic links between ontologies is essential for enabling interoperability across heterogeneous and interdisciplinary knowledge domains. However, manually curating such links is difficult to scale. To address this challenge, we propose an end-to-end framework for ontology network construction that automates the discovery and generation of both intra-domain and inter-domain relationships. Our approach combines domain-adapted DistilBERT embeddings for dense contextual representation, clustering-based pre-filtering to reduce the candidate search space, and GPT-4o-driven relationship generation via iterative prompt engineering to produce semantically rich, interpretable links. Applied to ReproduceMeON - a network of 33 ontologies spanning machine learning, microscopy, computational science, and experimental workflow - the pipeline reduces approximately 800k raw concept pairs to 95k high-quality candidates. Human expert validation of 429 generated relationships by two independent annotators yields an overall precision of 80.19% (91.49% on high-certainty annotations) and an F1 of 0.890, with substantial inter-annotator agreement. Comparative experiments against five similarity-based baselines, including Sentence-BERT, show a substantial performance gap (best baseline F1 = 0.581), while an ablation study demonstrates that similarity-based methods alone fail to discriminate valid from invalid relationships (AUC approx 0.5) on the filtered candidate set. These findings highlight the necessity of LLM-based reasoning over concept roles and domain semantics for accurate relationship construction.

1. Introduction

The introduction argues that ontology networks need typed semantic links across domains, but existing alignment, embedding, and LLM approaches leave key parts of link generation unaddressed. It presents an end-to-end pipeline and reports stronger evaluation results than similarity-based baselines.

1. Introduction

  • Ontology networks link multiple ontologies through typed semantic relationships, supporting cross-domain knowledge integration where a single monolithic ontology is impractical.The introduction distinguishes intra-domain links between ontologies in the same domain from inter-domain links across distinct domains.Source: §1 · p. 1
  • Classical alignment tools identify equivalences, embeddings struggle to distinguish valid from invalid high-similarity pairs, and prior LLM approaches lack systematic pre-filtering and role-based materialization.The introduction reports that all five similarity methods, including Sentence-BERT, achieve AUC ≈0.5 on the pre-filtered candidate set.Source: §1 · p. 1
  • F1 = 0.890 and AUC-ROC = 0.927 on a balanced 929-pair set; Sentence-BERT’s F1 = 0.581, a 30.9-percentage-point gap.The pipeline’s overall precision is 80.19%, rising to 91.49% on high-certainty annotations.Source: §1 · p. 2
  • The proposed pipeline combines domain-adapted DistilBERT, 90th-percentile clustering-based filtering, and GPT-4o relationship generation with structured prompts and role-based mapping.It reduces approximately 800k raw pairs to 95k candidates; the contributions also include scientific justifications and workflow-oriented roles.Source: §1 · p. 2

2. Related Work

Prior ontology-network approaches rely on manual curation, rule-based alignment, or embeddings, while LLM integration for structured, role-aware links remains underexplored. The paper positions its contribution as a combined end-to-end approach for typed links rather than equivalence matching.

2. Related Work

  • Manual curation and rule-based alignment are labor-intensive, domain-specific, and often struggle to scale to dynamic or interdisciplinary integration.Traditional methods include schema matching, logical alignment, and ontology mediation.Source: §2 · p. 2
  • Embedding methods such as RDF2Vec and OWL2Vec* [5] improve scalability but can miss semantic nuances requiring contextual or domain-specific interpretation.OWL2Vec* uses structural and contextual ontology features to generate embeddings.Source: §2 · p. 2
  • LLM studies address ontology development, relation extraction, and knowledge graphs, but structured, role-aware generation of intra- and inter-domain links remains underexplored.Prompt engineering for semantically meaningful and structurally usable ontology relationships is also described as early-stage work.Source: §2 · pp. 2–3
  • This work targets typed functional, causal, and other associations between non-equivalent concepts, not formal OWL axioms or equivalence mappings.It combines embeddings, pre-filtering, structured LLM prompting, and role-based materialization in one validated pipeline for a multi-domain ontology network.Source: §2 · p. 3

3. Methodology

The methodology defines typed, justified link discovery as a six-stage process: extract ontology concepts, generate contextual embeddings, filter candidate pairs, generate links with an LLM, map roles, and evaluate results.

3. Methodology

  • The task is to discover typed, justified associations between ontology classes, excluding equivalence mappings and focusing on links such as functional, causal, compositional, and instrumental relations.A link is valid when a domain expert confirms its relationship in the concepts’ ontology-domain context.Source: §3 · p. 3
  • The six-stage pipeline extracts class triples, fine-tunes DistilBERT, filters candidates, generates GPT-4o links, maps roles, and conducts expert validation and evaluation.The filtering stage uses a 90th-percentile cosine threshold to reduce 800k pairs to 95k.Source: §3 · p. 3

3.1. Ontology Parsing & Filtering

  • Ontology parsing retains class-type RDF triples and removes literals and non-class data, while labels, comments, and annotations add context for disambiguating domain-specific meanings.The example distinguishes the meaning of “process” in machine-learning and experimental settings.Source: §3.1 · pp. 3–4

3.2. Contextual Embedding Generation

  • RDF triples enriched with labels and comments become context sentences for DistilBERT fine-tuning, which uses masked language modeling to learn subject-object semantics.The dataset is split into 80% training, 10% validation, and 10% test; the final configuration uses 25 epochs and a batch size of 16.Source: §3.2 · p. 4
  • Mean pooling produces sentence and subject/object embeddings, which KMeans organizes into semantically coherent groups for downstream pair identification.Cosine similarities are then computed only within clusters to focus comparisons and reduce computational overhead.Source: §3.2 · p. 4, §3.3 · p. 4

3.3. Similarity & Cluster Filtering

  • A dynamic 90th-percentile within-cluster similarity threshold adapts to each cluster’s score distribution, reducing the candidate search space while retaining semantically useful pairs.A fixed 0.97 threshold yielded 30% fewer candidates with no measurable downstream precision gain.Source: §3.3 · pp. 4–5

3.4. LLM Relationships Generation

  • GPT-4o evaluates candidate pairs using similarity, concept roles, dependencies, and real-world interactions, then returns typed links with justifications and provenance.Nine relationship types cover hierarchical, functional, causal, temporal, spatial, compositional, comparative, transformational, and instrumental links.Source: §3.4 · p. 5

3.4.1. Prompt Structure and Example

  • Structured prompts give GPT-4o concept and ontology context, domains, and similarity scores, while requiring a specified JSON output with a relationship type and scientific justification.Iterative refinement added contextual cues, task constraints, chain-of-thought decomposition, and few-shot examples to reduce vague outputs.Source: §3.4.1 · p. 5, §3.4.2 · p. 6

4. Experimental Evaluation

The experimental evaluation asks whether large language models and contextual embeddings can generate accurate, semantically meaningful ontology-network relationships and describes the implementation environment.

4. Experimental Evaluation

  • The evaluation addresses how LLMs and contextual embeddings can generate accurate, semantically meaningful relationships for ontology-network development.The section introduces its experimental setup, datasets, and results.Source: §4 · p. 6

4.1. Experimental Setup

  • Experiments ran on Google Colab with an NVIDIA T4 GPU, using Transformers for DistilBERT fine-tuning and PyTorch for training and embedding generation.Source: §4.1 · p. 6
  • The implementation used Scikit-learn for cosine similarity and KMeans, RDFlib for ontology parsing, and the OpenAI GPT-4o API for relationship generation.Source: §4.1 · p. 7

4.2. Datasets

The evaluation covers computational, experimental workflow, machine learning, and microscopy ontologies, with provenance concepts included within experimental workflow rather than separately partitioned. RDF triples are classified by ontology domain to describe the dataset.

4.2. Datasets

  • The four study domains are computational, experimental workflow, machine learning, and microscopy; provenance concepts remain within experimental workflow rather than a separate partition.The dataset includes alignments between ontologies and provenance concepts captured through experimental-workflow triples.Source: §4.2 · p. 7
  • 46,244 unique subjects, 71,426 unique objects, and 214 unique predicates were extracted from ReproduceMeON RDF triples.The triples were classified by ontology domain, and Table 1 presents their distribution.Source: §4.2 · p. 7, Table 1 · p. 7
  • More than 20 individual relation types each contribute under 2%, including Processes, Applies, Extends, Controls, Measures, Depends On, and Precedes.Table 2 presents the distribution of relationship types across generated links.Source: §4.2 · p. 7, Table 2 · p. 7

4.3. Results

The pipeline generates typed links across and within domains, including an example Functional link from Microscopy to Computational concepts. Its link distribution includes both common relationship types and domain-specific variation.

4.3. Results

  • The pipeline identifies “Activity” in Microscopy as enabling “Objective” in Computational, labeling the inter-domain link as Functional.Table 2 reports relationship-type distributions across generated links.Source: §4.3 · p. 7

4.3.1. Diversity and inter/intra-domain balance

  • Enables accounts for 22.84% of links and Requires 13.75%; Functional relationships dominate all four domains, ranging from 56.13% in Microscopy to 84.67% in ML.Instrumental links are prominent in Microscopy (30.32%) and Computational (19.15%).Source: §4.3.1 · p. 7
  • The pipeline produces 51.3% inter-domain links and 48.7% intra-domain links, while Experimental Workflow has the broadest variety of relationship types.Compositional (11.63%), comparative (9.30%), and hierarchical (9.30%) links also contribute in Experimental Workflow.Source: §4.3.1 · pp. 7–8, Figure 3 · p. 8

4.4. Quantitative Baseline Comparison

Against five similarity-based baselines, the LLM pipeline achieves higher F1 and precision on the 929-pair evaluation set. The comparison targets typed, justified relationship validity, not equivalence mapping by ontology alignment systems.

4.4. Quantitative Baseline Comparison

  • The comparison tests whether similarity alone judges relationship validity; it does not claim superiority over LogMap or AML equivalence mappings.Comparison with other LLM-based relationship generators is left for future work because no publicly available system produces the same typed, justified, role-mapped output format.Source: §4.4 · p. 8
  • The evaluation set contains 344 valid annotated relationships, 85 invalid annotated relationships, and 500 randomly generated cross-domain negatives.The LLM pipeline predicts valid for all 429 annotated pairs and invalid for the 500 random pairs; randomly sampled negatives are easier than real-world hard negatives.Source: §4.4 · p. 8
  • F1 = 0.890 for the LLM pipeline versus F1 = 0.581 for Sentence-BERT, the best embedding baseline, on the 929-pair evaluation set.The pipeline also achieves Precision = 0.802, a +23.2 pp difference; Table 3 gives the baseline comparison and Figure 4 shows ROC and precision–recall curves.Source: Table 3 · p. 8, §4.4 · p. 8, Figure 4 · p. 9

4.5. Ablation Study: Why Similarity Alone Is Insufficient

On the 429 pre-filtered pairs, similarity scores alone do not distinguish valid from invalid relationships. The embedding filter nevertheless reduces the candidate set for the LLM stage.

4.5. Ablation Study: Why Similarity Alone Is Insufficient

  • AUC ≈0.5 for every similarity method on the 429 pre-filtered pairs, indicating random-level validity discrimination among high-similarity candidates.The pairs had already passed the 90th-percentile similarity filter.Source: Table 4 · p. 9, §4.5 · p. 9
  • The authors limit the ablation conclusion to similarity scores in this setting; it does not rule out graph-based, symbolic, or supervised alternatives.They identify contextual reasoning about concept roles, real-world dependencies, and domain-specific semantics as missing from similarity metrics.Source: §4.5 · p. 9
  • DistilBERT clustering and the 90th-percentile threshold reduce approximately 800k candidate pairs to ≈95k, an 88% reduction.The text describes removing the embedding pre-filter as computationally infeasible.Source: §4.5 · p. 9

4.6. Human Annotation and Validation Results

Expert review found 80.19% precision among 429 generated relationships, with substantial agreement on dual-annotated pairs. Precision varied by domain and relationship type, while similarity validation and exhaustive recall assessment remained limited.

4.6. Human Annotation and Validation Results

  • Cohen’s κ was 0.723 with 90.5% agreement across 274 dual-annotated pairs; the 155 Microscopy pairs had single-annotator coverage only.Computational agreement reached κ = 0.817, while Machine Learning and Experimental Workflow reached κ = 0.671 and 0.720, respectively.Source: §4.6 · p. 10, Table 5 · p. 10
  • The NLI validator gave near-identical entailment scores to valid and invalid pairs, 0.976 versus 0.970, and Cohen’s κ = 0.0.The authors caution that fluent justifications can sound persuasive regardless of correctness, so domain expertise remains necessary for validation.Source: §4.6 · pp. 10–11
  • Recall cannot be computed without an exhaustive gold standard; the 429 validated pairs across four domains and multiple ontology sources provide only indicative coverage.Precision is therefore reported as the primary metric.Source: §4.6 · p. 11
  • 344 of 429 generated relationships were validated as correct, yielding 80.19% overall precision; the 91.49% high-certainty precision reflects less ambiguous cases.The high-certainty figure covers 376 annotations and should be interpreted cautiously because those cases are less ambiguous by definition.Source: §4.6 · p. 11
  • 84.58% intra-domain versus 75.25% inter-domain precision; causal and comparative relationships reached 100%, hierarchical 95.24%, and instrumental 65.22%.Figure 6 breaks precision down by domain and relationship type.Source: §4.6 · p. 11
  • Among 85 invalid links, inter-domain surface-term overlap often masked conceptual incompatibility, and instrumental errors frequently confused tool roles with process roles.The latter confusion was especially noted across Machine Learning and Experimental Workflow concepts.Source: §4.6 · p. 11

4.7. Limitations

The evaluation has limits in annotation coverage, negative sampling, domain scope, repeatability, and independent assessment of role assignments.

4.7. Limitations

  • Only 274 of 429 relationships received dual annotation; all 155 Microscopy pairs had single-annotator coverage, with Microscopy-specific κ left for future work.The second annotator covered three domains: Machine Learning, Computational, and Experimental Workflow.Source: §4.7 · p. 11
  • The balanced evaluation’s 500 randomly generated negatives may inflate F1 because they are easier to reject than hard negatives; pipeline-output precision is not subject to this caveat.Source: §4.7 · p. 12
  • Experiments cover only ReproduceMeON, and GPT-4o’s stochastic outputs may vary across runs, but output variance and prompt sensitivity were not measured.Evidence of generalizability to other domains or ontology sizes and annotation densities is lacking.Source: §4.7 · p. 12
  • The role taxonomy is grounded in established Ontology Design Patterns, but the correctness of its role assignments has not been independently evaluated.Competency-question testing is identified as future work.Source: §4.7 · p. 12

5. Conclusion

The paper concludes that an end-to-end pipeline combining fine-tuned DistilBERT, clustering-based candidate selection, iterative GPT-4o prompting, and role-based materialization can discover intra- and inter-domain links. On ReproduceMeON, expert validation and baseline and ablation comparisons support its relationship-generation performance and the need for reasoning beyond similarity.

5. Conclusion

  • The pipeline achieved 80.19% overall precision (91.49% on high-certainty annotations), F1 = 0.890, and exceeded the best embedding baseline, F1 = 0.581, by over 30 percentage points.Expert validation covered 429 generated relationships.Source: §5 · p. 12
  • Among pre-filtered high-similarity pairs, all similarity methods achieved AUC ≈0.5; the authors attribute the precision gain to LLM reasoning about concept roles and domain dependencies.The result indicates that semantic similarity alone did not distinguish valid from invalid relationships in this candidate set.Source: §5 · p. 12
  • Agreement on 274 dually annotated pairs reached Cohen’s κ = 0.723, which the authors characterize as substantial and as confirming annotation-task reproducibility.The pipeline was evaluated across computational, machine-learning, microscopy, and experimental-workflow domains.Source: §5 · p. 12
  • Across four domains, pre-filtering reduced 800k raw candidate pairs to 95k high-confidence pairs, a step the authors describe as essential for computational feasibility.The pipeline combines fine-tuned DistilBERT embeddings, clustering-based candidate selection, iterative GPT-4o prompt engineering, and role-based relationship materialization.Source: §5 · p. 12
  • Future work targets open-source instruction-tuned LLMs, recall measurement with a curated gold-standard dataset, provenance-domain integration, and workflow-generation and SPARQL-reasoning extensions.These are proposed directions, not evaluated results.Source: §5 · p. 12

Related papers

Loading 2610.01393v1…