Benchmarking Automated Knowledge Graph Construction from Semi-Structured Data

1 Introduction

• The paper defines semi-structured KG construction with required and optional inputs to reflect realistic scenarios.This task definition is presented as the basis for evaluation and comparison.Source: §1 · p. 2
• The work reviews existing evaluation frameworks and extends them to assess KG creation and downstream usefulness.The framework supports systems that predict mappings or produce RDF data.Source: §1 · p. 2
• The framework evaluates mappings, materialized KGs, and downstream task usefulness across two ontology-selection scenarios.It scores syntactic validity, semantic accuracy in the restricted scenario, consistency, conciseness, completeness, and pragmatic quality.Source: §1 · p. 2
• The benchmark provides 10 expert-curated real-world datasets from domains including ecology, biodiversity research, genomics, copolymer chemistry, and history.Each dataset includes base data, competency questions and gold answers, target ontologies, and gold mappings.Source: §1 · p. 2
• The authors showcase the benchmark with two recent reference systems across multiple evaluation scenarios and discuss the results and implications.Source: §1 · p. 2

2 Task Definition

• The task definition reflects deployment conditions by allowing users to supply context files that may also support ablation studies.The paper argues that users can provide additional context to increase the probability of task success.Source: §2 · p. 3
• The task takes research data and metadata, competency questions with gold answers, and target ontologies as required inputs, with intent files, ontology documentation, and research papers optional.Questions requiring links to unavailable external sources are omitted.Source: §2 · p. 3
• Evaluation covers mappings, the materialized KG, and KG usefulness for downstream tasks.The stages ask whether mappings are valid, generated RDF is consistent and complete, and the KG can answer competency questions.Source: §2 · p. 3
• For the first benchmark release, pragmatic quality measures whether the KG can answer the competency questions used to create it.The paper treats these questions as evidence of how well a KG represents its source data and domain knowledge.Source: §2 · p. 3, D1 – Syntactic Validity · p. 4
• The framework reuses intrinsic linked-data quality dimensions and adapts pragmatic quality to assess usefulness for a downstream task.The dimensions include syntactic validity, semantic accuracy, consistency, conciseness, completeness, and pragmatic quality.Source: §2 · p. 4, D1 – Syntactic Validity · p. 4

• SemTab evaluates table-to-KG matching tasks but not explicit declarative mappings; its hierarchical prediction scoring is reused for comparing predicted KGs with reference KGs.Source: §3 · p. 4
• RODI evaluates relational-database-to-ontology mapping through query answers, but excludes complex transformations and XML/JSON, and does not directly assess mappings or graph content beyond query answering.Source: §3 · pp. 4–5
• KGrEaT scores KGs on downstream tasks including classification, regression, clustering, similarity, relatedness, analogies, and recommendation; KG question answering was not yet integrated.Source: §3 · p. 5
• BLINKG [6] uses tabular mapping predictions, similarity-based scoring, and a feature taxonomy; this work adds metrics for similarity-scoring false positives and ontology documentation to distinguish mapping errors from unfamiliarity with ontology patterns.Source: §3 · pp. 5–6

4.1 Evaluation Pipeline

• Systems can submit mapping tables or RDF; pipeline modules reconstruct mappings, compile mappings into RDF, and generate and execute SPARQL queries for competency questions.The evaluation stages score mappings, the materialized KG, and KG usefulness through competency-question answering.Source: §4.1 · p. 7

4.2 Quality Dimensions per Stage and Evaluation Metrics.

• Syntactic validity checks mapping references and components separately from RDF serialization, which is strictly validated with Apache Jena RIOT.The reported validation diagnostics include warnings, errors, and fatal errors; fatal parsing errors may stop validation before all errors are counted.Source: D1 Syntactic Validity · pp. 7, 9
• The first pipeline release measures semantic accuracy only when benchmark target ontologies and reference mappings are available; it omits this measure for arbitrary-ontology submissions.Without target ontologies or reference mappings, the authors say semantic accuracy cannot yet be scored deterministically.Source: D2 Semantic Accuracy · pp. 10, 12
• Semantic accuracy compares mappings and generated assertions with expert-curated references through mapping-, assertion-, entailment-, and entity-neighborhood metrics.Entity-centered neighborhood accuracy macro-averages local F1 over one-hop incoming and outgoing assertions; unmatched entities receive zero, so every entity has equal weight.Source: D2 Semantic Accuracy · pp. 10, 12
• Consistency assesses mappings and materialized RDF against ontology axioms independently of gold mappings, distinguishing compatible, incompatible, and unknown mapping cases.The framework checks logical consistency with an OWL reasoner; under the open-world assumption, lack of entailment alone is not a violation.Source: D3 Consistency · pp. 12–13
• Conciseness extends beyond redundant schema elements and entity representations to redundancy from mapping decisions, identifiers, and generated property values at mapping and materialized-KG stages.Source: D4 Conciseness · pp. 13–14

4.2 Quality Dimensions per Stage and Evaluation Metrics

• CN1–CN5 capture schema redundancy, duplicate identifiers, identifier collisions, duplicate generated facts, and equivalent property values across mappings (B1) and materialized KGs (B2).CN4 requires pre-deduplication mapping provenance, and mapping-level conciseness is fully observable only for systems that provide mapping predictions.Source: D4 Conciseness · pp. 14–15
• Completeness assumes a closed world over benchmark source data: CM1 checks required source-field use, while CM2 and CM3 assess represented source information and entities in the materialized KG.High completeness does not by itself establish semantic accuracy, conciseness, or downstream usefulness.Source: D5 Completeness · pp. 16, 18
• Pragmatic quality compares generated-KG answers with gold competency-question answers using correctness, completeness, F1, macro/micro aggregates, exact satisfaction, hierarchy-aware F1, and difficulty strata.Its scope is the benchmark's competency questions; high scores do not imply usefulness for downstream tasks they do not cover.Source: D6 Pragmatic Quality · p. 18

• The benchmark contains 10 hand-curated, real-world datasets across ecology, biodiversity research, genomics and metabolomics, copolymer chemistry, and history.Source: §5 · p. 20
• The appendix and repository provide complete documentation, including KG schemas, reference mappings, competency questions, gold answers, and target-ontology information.Source: §5 · p. 20
• The section presents an overview of each dataset and directs readers to the appendix or repository for full documentation.Source: §5 · p. 20

7.1 Implications and Limitations

• The pipeline evaluates mappings alongside their pragmatic usefulness and can quantify the impact of optional context files and their combinations.Optional inputs include data-intent sheets, papers or texts, and ontology tutorials.Source: §7.1 · p. 20
• Modular evaluation supports varied strategies and materialized-KG entry for CQ, consistency, completeness, or schema checks, enabling pragmatic comparisons across schemas.The framework does not provide general semantic evaluation of mapping choices across unrestricted ontology sets.Source: §7.1 · p. 20
• Competency-question quality, including coverage of input data, significantly influences evaluation scores and generated KGs.Source: §7.1 · pp. 20–21
• The compiler translates BLINKG tabular mappings into RML mappings, eliminating syntactically incorrect RML mappings as an error source.Source: §7.1 · p. 21
• Ontology documentation may leak into LLM training data and confound mapping performance, while lesser-known ontologies may be underrepresented.Source: §7.1 · p. 21
• The benchmark may not exhaustively represent real-world domains and scenarios; dataset difficulty, ontology counts, and reference-mapping complexity are difficult to gauge.Source: §7.1 · p. 21
• The amount of context may improve task performance, but some users and open LLMs may have limited computational resources to use it.Source: §7.1 · p. 21

• The research was supported by a Flexpool grant from the German Centre for Integrative Biodiversity Research Halle-Jena-Leipzig (iDiv).Source: §9 · p. 21
• The authors used ChatGPT for paraphrasing, style improvements, and grammar and spelling checks, then reviewed and edited the content.Source: Declaration on Generative AI · p. 21
• The authors declare no competing interests relevant to the article’s content.Source: Declaration on Generative AI · p. 22