PaperlayerView exact revision on arXiv

Source-linked AI summary

CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions

Jiayi Li, Ziyuan Wang, Daniel Garijo, María Poveda-Villalón

arXiv:2609.26029v1cs.AI

TL;DR

Evaluating LLM-generated ontologies from competency questions is difficult because studies use heterogeneous tasks, CQ provenance is often limited, and common metrics can miss structural and logical adequacy. CQ4OE builds CQ-aligned gold standards and complementary tasks, then evaluates nine LLMs. The experiments find that models recover explicit vocabulary more reliably than ontology structure, with reasoning-based closure offering limited recovery when hierarchy is missing.

  • Problem

    LLM ontology-generation evaluations use heterogeneous task formulations, often lack CQ-specific links to required terms and axioms, and may not assess property modeling, logical constraints, or reasoning behavior.

    Source: §1 · p. 2

  • Method

    CQ4OE provides CQ-aligned gold ontologies and provenance with two evaluation tasks, and compares nine LLMs using zero-shot, iterative, and multi-agent generation strategies.

    Source: §1 · p. 2, §6 · p. 12

  • Results

    Experiments across six ontologies show that models recover explicit vocabulary more reliably than ontology structure, while reasoning-based closure provides limited recovery when hierarchy is missing.

    Source: §7 · p. 16

  • Takeaways & Limitations

    The results support provenance-aware, multidimensional evaluation of LLM-generated ontologies, with CQ4OE proposed as a common evaluation basis for future research.

    Source: §7 · p. 16

    Ontology-level metrics depend on term alignment, which can fail for short property labels or semantically close but non-equivalent terms; alignment thresholds are applied uniformly across six heterogeneous ontologies.

    Source: §6 · p. 15

Abstract

from arXiv · show

Ontology generation from Competency Questions (CQs) is a central yet labor-intensive phase of Ontology Engineering. While large language models (LLMs) offer promising automation capabilities, current evaluations remain fragmented. Task formulations are heterogeneous, gold standards often lack fine-grained CQ provenance, metrics conflate lexical overlap with structural and logical adequacy, and reference ontologies are not always explicitly designed around the evaluation CQs. Here, we address these limitations with CQ4OE, a benchmark for the systematic and reproducible evaluation of LLM-based ontology generation from CQs. For each ontology in the benchmark, we build a CQ-driven gold OWL ontology with explicit provenance linking each CQ to the classes, properties, and axioms required to answer it. From this resource, we define two complementary evaluation tasks. CQ2Term supports term-level evaluation of CQ-specific class and property prediction over 99 CQs, and CQ2Onto supports ontology-level evaluation over 118 CQs, including hierarchy, property modeling, and axiom-level structure. We demonstrate CQ4OE with experiments using nine LLMs under zero-shot, iterative, and multi-agent generation strategies, showing that LLMs recover explicit vocabulary terms more reliably than creating ontologies, particularly in property modeling, hierarchy construction, and axiom generation.

1 Introduction

Ontology engineering remains expertise-intensive, while LLM evaluations use heterogeneous tasks, weak CQ provenance, and metrics that can miss structural and logical errors. CQ4OE addresses these gaps with CQ-aligned gold standards, complementary evaluation tasks, and an automated reporting pipeline, then evaluates nine LLMs under multiple generation strategies.

1 Introduction

  • Existing evaluations vary in task, input/output specification, and modeling depth, often lack CQ-specific term and axiom links, and may miss property, constraint, and hierarchy errors.These differences make direct comparison and assessment of whether an ontology satisfies CQ requirements difficult.Source: §1 · p. 2
  • CQ4OE provides CQ-aligned gold standards for two complementary tasks, covering term-level prediction over 99 CQs and ontology-level evaluation over 118 CQs.CQ2Onto includes six ontologies and fine-grained CQ-to-axiom provenance.Source: §1 · p. 2
  • Its metrics assess term recovery, property characteristics, domain/range triples, TBox axioms, and hierarchy closure beyond surface vocabulary overlap.The framework evaluates whether generated ontologies are structurally and logically sound.Source: §1 · p. 2
  • An automated reporting pipeline traces matched and missing terms, axioms, CQ coverage, and reasoning-aware hierarchy recovery for generated ontologies.The reports connect evaluation outputs to specific ontology elements.Source: §1 · p. 2
  • The benchmark compares nine LLMs using zero-shot term prediction for CQ2Term and zero-shot, iterative, and multi-agent strategies for CQ2Onto.The strategies are specified separately for the two tasks.Source: §1 · p. 2

2 Related Work

Prior resources address neighboring tasks or evaluate generated ontologies without CQ-driven requirements, while studies use different CQ sets, references, and metrics. CQ4OE fills the gap with a reusable benchmark that evaluates requirement satisfaction across multiple ontology dimensions.

2 Related Work

  • Prior CQ-to-ontology approaches use different CQ sets, reference ontologies, and metrics, while full reference ontologies may include knowledge beyond the input CQs.MASEO generates OWL ontologies from CQs with provenance links but does not release a reusable evaluation resource.Source: §2 · p. 3
  • Existing resources target ontology matching, biomedical question answering, missing-axiom identification, CQ quality, or qualitative ontology assessment rather than CQ-driven ontology construction.CORAL provides requirements and CQs but does not map each CQ to the terms or axioms needed to answer it.Source: §2 · p. 3
  • CQ4OE supplies a reusable benchmark with shared CQs, CQ-aligned gold standards, and metrics spanning term recovery, property characteristics, domain/range relations, TBox axioms, and hierarchy closure.Source: §2 · p. 3

3 CQ4OE Benchmark Dataset Construction

CQ4OE constructs two CQ-aligned gold standards from selected source ontologies, using a reviewed annotation workflow to link CQs to terms and required axioms. The resulting tasks cover explicit term prediction and CQ-relevant ontology structure.

3 CQ4OE Benchmark Dataset Construction

  • CQ4OE creates CQ-aligned gold standards for term prediction and ontology-level evaluation, recording CQ provenance for terms and required axioms.Source: §3 · p. 3

3.1 Source ontologies

  • Six publicly redistributable source ontologies span three CQ-count tiers and varied hierarchy shapes, from nearly flat ODRL to deeply nested SWO.Selection also required established OE use and documented requirements with associated CQs.Source: §3.1 · pp. 3–4

3.2 Annotation methodology

  • The annotation workflow filters CQs, labels explicit, implicit, and derived terms, and adds CQs when needed to cover core terms; 8 additions yield 118 CQs.Of the 118 CQs, 101 (85.6%) were accepted without change, and the 17 revised items were concentrated in more complex ontologies.Source: §3.2 · pp. 5–6, Figure 1 · p. 6

3.3 CQ2Term and CQ2Onto gold standards

  • CQ2Term retains 99 CQs with explicit terms, while CQ2Onto links CQ-relevant terms and only those TBox axioms whose removal would make a CQ unanswerable.Each annotation item receives initial labeling, expert review, and adjudication when needed.Source: §3.2 · p. 5, §3.3 · p. 6

4 Benchmark Evaluation Methodology

CQ4OE evaluates generated ontologies by aligning predicted and gold terms before measuring quality at multiple modeling depths, from CQ-specific term recovery to axioms and hierarchy. Its methodology combines term-alignment metrics with ontology-level structural and reasoning-aware evaluation.

4 Benchmark Evaluation Methodology

  • The methodology first aligns predicted and gold terms, then evaluates ontology quality beyond lexical overlap at multiple modeling depths.The section defines term alignment in 4.1 and metrics for CQ2Term and CQ2Onto in 4.2 and 4.3.Source: §4 · p. 6

4.1 Term alignment aggregation and selection

  • Five similarity methods are aggregated by averaging the top three non-hard scores, with exact hard matches overriding the aggregate; thresholded pairs are greedily selected one-to-one.Class and property thresholds are 0.6 and 0.7, respectively, with the higher property threshold intended to reduce spurious matches on short formulaic labels.Source: §4.1 · p. 7

4.2 CQ2Term Evaluation

  • CQ2Term measures globally aligned class and property terms, and its CQ-conditioned coverage checks whether recovered terms are assigned to the CQs that require them.At-least-one, mean, and full coverage capture complementary aspects of term recovery; high overall vocabulary recovery may still leave individual CQs unanswered.Source: Term-level metrics · p. 8

4.3 CQ2Onto Evaluation

  • CQ2Onto assesses term recovery, property characteristics, domain/range triples, TBox axioms, and inferred hierarchy closure, with global and alignment-conditioned views for structural targets.It also reports CQ-level axiom coverage, including closure recovery only for eligible missed hierarchy axioms that decompose into atomic subclass or subproperty pairs.Source: Ontology-level metrics · p. 9, §4.3 · p. 9, CQ-conditioned coverage · pp. 9–10
  • Per-run reports trace matched, closure-recovered, and missed axioms and label recovered cases by inference type, with shortest-path explanations for chain-inferred cases.Reports also include evaluation scores, term alignments, and per-axiom traces.Source: Evaluation report · p. 10

5 Experimental Setup and Results

The experiments benchmark nine LLMs across CQ2Term and CQ2Onto, finding stronger recovery of vocabulary than ontology structure and substantial gaps in CQ-level completeness.

5 Experimental Setup and Results

  • Experiments cover 99 CQs for CQ2Term and 118 CQs across six ontologies for CQ2Onto, using nine LLMs and zero-shot term prediction plus three CQ2Onto strategies.The CQ2Onto strategies are zero-shot, iterative, and multi-agent generation.Source: §5 · p. 10

5.1 Results

  • CQ-conditioned coverage averages 89.2% at-least-one but only 23.5% full, showing that strong global vocabulary recovery does not imply CQ-level completeness.Water reaches 70.6% global F1 but 0% full coverage for every model; Wine reaches 100% at-least-one but 0% full coverage.Source: §5.1 · p. 11
  • CQ2Term global term F1 ranges from 59.1% (DeepSeek V3.2) to 66.5% (DeepSeek V4-Pro), while CQ2Onto class F1 averages 59.7% versus property F1 31.8%.Source: §5.1 · p. 11
  • CQ2Onto closure F1 averages 16.7%, and Triple-AC 36.4% versus Triple-G 12.4% and Axiom-AC 35.3% versus Axiom-G 15.0%.AWO is excluded from triple aggregation because its gold range is a complex anonymous OWL expression.Source: §5.1 · p. 11
  • Iterative prompting averages 8.6 closure pairs versus 4.9 for zero-shot and 4.7 for multi-agent; multi-agent raises Axioms-Mean from 18.3% to 24.2%.Closure rescue raises Closure-Mean by 1.4 points but leaves Closure-Full unchanged; it yields no measurable improvement on ODRL, Water, VGO, or SWO.Source: §5.1 · pp. 11–12
  • Models rarely generate sufficient hierarchy chains, paraphrase or alias properties more often than classes, and recover some CQ-required axioms but rarely all, leaving Axiom-Full near 2%.VGO and SWO predicted ontologies have 0.6 and 0.7 closure pairs against gold 12 and 8, respectively.Source: §5.1 · p. 12

6 Discussion

CQ4OE links competency questions to the terms and axioms required to answer them, addressing the difficulty of evaluating generated ontologies against CQ requirements rather than broad domain knowledge. Its discussion presents complementary evaluation tasks and traceable reporting, while noting limitations in term alignment, closure rescue, and contamination control.

6 Discussion

  • Full reference ontologies can include knowledge unrelated to evaluated CQs, making it difficult to distinguish requirement satisfaction from general domain overlap.CQ4OE links terms and TBox axioms to the CQs that require them.Source: §6 · p. 12
  • CQ4OE preserves provenance from individual CQs to classes, properties, and axioms, a level of CQ-to-axiom provenance the authors say no existing benchmark provides.This makes the benchmark requirement-driven rather than relying on full reference ontologies as undifferentiated gold standards.Source: §6 · pp. 12–13
  • CQ2Term assesses explicit CQ vocabulary, whereas CQ2Onto assesses implicit and derived terms and their expression in coherent ontology structure.CQ2Onto covers property characteristics, domain/range relations, TBox axioms, and hierarchy under reasoner-derived closure.Source: §6 · pp. 13–14
  • CQ4OE reports trace metrics to specific gold and predicted axioms, helping distinguish missing vocabulary, unstable property modeling, shallow hierarchies, and incomplete CQ coverage.Its Markdown reports and CSV exports support diagnosis at individual-CQ and axiom levels.Source: §6 · p. 15
  • The benchmark is designed for reuse across domains, models, prompting strategies, repair pipelines, and human-in-the-loop workflows.Users can add a domain under a triple-review, adjudication-based annotation protocol.Source: §6 · p. 15
  • Ontology-level metrics depend on term alignment, so alignment errors propagate; uniform thresholds across six heterogeneous ontologies can still miss short labels or close but non-equivalent terms.The thresholds are configurable, and users can inspect and correct alignments before rerunning evaluation.Source: §6 · p. 15
  • Closure rescue quantifies only hierarchy axioms decomposable into atomic subclass or subproperty pairs, excluding some complex expressions from evaluation.The authors identify partial credit for sub-expressions within complex axioms as an open problem.Source: §6 · pp. 15–16
  • Public source ontologies and published CQs may appear in pretraining data, although future benchmark versions plan private or post-cutoff ontologies for contamination-controlled evaluation.The authors also note that manually curated CQ provenance is an additional annotation layer unlikely to have appeared in pretraining data.Source: §6 · p. 16

7 Conclusion

CQ4OE links each competency question to the terms and axioms needed to answer it and provides CQ2Term and CQ2Onto for complementary evaluation. Experiments across nine LLMs and six ontologies show more reliable recovery of explicit vocabulary than ontology structure, with reasoning-based closure offering limited recovery when hierarchy is missing.

7 Conclusion

  • CQ4OE links each CQ to the terms and TBox axioms required to answer it through complementary tasks, CQ2Term and CQ2Onto.Its reports and CSV exports trace metrics to gold and predicted axioms for diagnosing why individual CQs are not satisfied.Source: §7 · p. 16
  • Across nine LLMs and six ontologies, explicit vocabulary is recovered more reliably than ontology structure, and performance varies more across domains than across models or strategies.Reasoning-based closure provides only limited recovery when the underlying hierarchy is missing.Source: §7 · p. 16
  • The authors identify provenance-aware, multidimensional evaluation as a need and plan to add domains, refine metrics, and incorporate private or post-cutoff ontologies.The planned additions are intended to support contamination-controlled evaluations.Source: §7 · p. 16

Resource Availability Statement

CQ4OE is openly available under Apache 2.0, with its benchmark resources and evaluation materials hosted in the project repository and archived on Zenodo and HuggingFace.

Resource Availability Statement

  • The Apache 2.0 resource includes CQ2Term and CQ2Onto gold standards for six ontologies, the evaluation pipeline, and run-level reports.The paper lists the project repository, Zenodo, and HuggingFace as access points.Source: Resource Availability Statement · p. 17

Declaration of use of Generative AI

The authors report using Claude to improve manuscript clarity, grammar, and readability, then reviewing and editing the work and accepting responsibility for its content.

Declaration of use of Generative AI

  • Claude was used for clarity, grammar, and readability; the authors reviewed and edited the manuscript and take full responsibility for its content.The disclosure concerns manuscript preparation.Source: Declaration of use of Generative AI · p. 17
Loading 2609.26029v1…