PaperlayerView exact revision on arXiv

Source-linked AI summary

Ontology-Guided Knowledge Graph Construction from Structured Metadata and Climate Science Publications

Aayush Acharya, Qi Zhang, Eduard Dragut, Longin Jan Latecki

arXiv:2610.32725v1cs.CLcs.AI

TL;DR

Scientific literature is growing faster than manual curation, while comprehensive climate knowledge graphs integrating vocabularies with literature-derived knowledge remain scarce. The paper builds an evidence-grounded, ontology-guided extraction framework and applies it to climate publications. On 1,007 publications, it reports higher correctness and more validated triplets than KARMA.

  • Problem

    Comprehensive knowledge graphs integrating climate vocabularies with systematically extracted literature knowledge remain scarce.

    Source: §1 · p. 2

  • Method

    An agentic, ontology-guided pipeline extracts evidence-backed triplets, checks ontology conformance and evidence, and links entities to the knowledge graph.

    Source: §3.3 · p. 4, §3.5 · p. 5

  • Results

    17,141 triplets with 99.067% correctness versus KARMA’s 3,820 triplets with 92.277% correctness on the first 1,007 publications.

    Source: §4.3 · p. 7

  • Takeaways & Limitations

    The resulting climate KG connects canonical climate entities with literature-derived facts supported by text evidence.

    Source: §1 · p. 2

    Triplet-level recall is not computed because exhaustive gold triplets would be required for each open-ended scientific document.

    Source: Limitations · p. 10

Abstract

from arXiv · show

The rapid expansion of scientific literature far outpaces the speed of manual curation, creating a fundamental bottleneck in extracting structured knowledge from scientific texts. To address this, we introduce an agentic, ontology-guided framework for knowledge graph (KG) construction. Our framework utilizes specialized agents for document understanding, information extraction, ontology conformance, and entity linking. First, an LLM extracts factual knowledge in the form of triples with explicit textual evidence from research papers. Then the triplets undergo a multistage verification process to ground the triplets in the ontology. By enforcing schema constraints and requiring evidence in the form of text fragments, we ensure the data is both semantically consistent and traceable. In an agreement study over 1,000 extracted triplets, Gemini-2.5-flash achieves Cohen's $\kappa$ scores of 0.640--0.700 against three human annotators, matching the human--human agreement range of 0.616--0.689. We applied this framework to 2,000 scientific publications and construct a novel, ontology-grounded KG for climate science with 133,990 nodes and 755,308 relations; of the 23,327 triplets that the publications contribute after validation, 98.96\% are judged correct. The pipeline code and KG is publicly available https://github.com/aayushacharya/climatekg.

1 Introduction

The paper addresses the lack of comprehensive climate knowledge graphs integrating controlled vocabularies with literature-derived knowledge. It introduces evidence-grounded, ontology-guided extraction and reports a large climate KG alongside validation results.

Motivation

  • Climate vocabularies primarily describe datasets and metadata, while comprehensive integration with systematically extracted literature knowledge remains limited.Climate extraction is challenging because long-tail entities and implicit dependencies may not be stated explicitly.Source: §1 · pp. 1–2
  • Ontology-guided extraction encodes domain knowledge and constrains relation semantics to handle specialized climate terminology and implicit entity relationships.For example, interpreting a model-component relation requires identifying entity types and the ontology-defined predicate.Source: §1 · p. 2

Formulation

  • The paper formalizes domain-specific KG construction as extracting triples with explicit textual evidence, streamlining literature curation into verification.The framework requires textual evidence and automated validation of both each triplet and its evidence.Source: §1 · p. 2

Framework

  • The framework uses specialized LLM agents for document understanding, schema conformance, and hybrid entity linking, with climate vocabularies and evidence validation.Extracted relations are traceable to source text.Source: §1 · p. 2

Experiments

  • 98.96% of 23,327 validated triplets are judged correct, while Gemini-2.5-flash reaches Cohen’s κ of 0.640–0.700 against three annotators.The agreement is comparable to human–human agreement; verification costs roughly 1% of extraction.Source: §1 · p. 2

Climate KG

  • 133,990 nodes and 755,308 relations comprise a climate KG enriched with evidence-grounded relations from 2,000 publications.The publication corpus contributes 73,283 nodes and 298,747 relations.Source: §1 · p. 2

2 Problem Formulation

The formulation defines a domain ontology and publication corpus, then sets the objective of building a knowledge graph whose relations are grounded in textual evidence.

Problem Formulation

  • The domain ontology O = (C, R) comprises entity classes and typed relations with explicit domain and range constraints.Source: §2 · p. 2
  • The corpus D consists of scientific publications, and the initial graph comes from established, trusted scientific sources.The graph is updated iteratively.Source: §2 · pp. 2–3
  • The target KG is G = (V, E), with each edge representing a triple grounded in explicit textual evidence.For each document, the objective is to extract evidence-backed triples.Source: §2 · p. 2

3 Methodology

The methodology builds a climate-science KG around a two-level ontology and a targeted 2,000-paper corpus. Its extraction pipeline grounds triples in source evidence, checks ontology conformance, initializes canonical graph entities from ESGF metadata, and uses hybrid entity linking.

3.1 Ontology Construction

  • A two-level ontology combines OpenAlex-derived scholarly metadata entities with domain-specific classes added for screened, relevant publications.Climate resources provide standardized concepts and domain/range constraints, such as Model as the domain and Component as the range of hasComponent.Source: §3.1 · p. 3, §3.1.1 · p. 3

3.1.1 Domain-Specific Ontology

  • The climate domain ontology draws on ESGF metadata, GCMD, CMIP6 Controlled Vocabularies, and GeoNames, with domain and range constraints serving as soft structural priors.The ontology was constructed with domain-expert input.Source: §3.1.1 · p. 3

3.2 Scientific Publication Corpus Collection

  • The corpus contains 2,000 papers: 1,007 selected through citation mining and relevance filtering, plus 993 recent candidates retrieved using OpenAlex concepts.The seed set drew on IPCC Sixth Assessment Report references and CMIP6 publications.Source: §3.2 · p. 3

3.3 Ontology-Guided IE Pipeline

  • The pipeline extracts triples with evidence, checks ontology conformance and source grounding, then validates whether the evidence supports each triple.Conformance and evidence grounding are independent checks; conformance runs first because it is cheaper and removes the largest share of candidates.Source: §3.3 · pp. 3–4, Figure 1 · p. 4

3.4 Graph Initialization

  • The KG starts from more than four million ESGF metadata records, deterministically mapped to ontology-aligned entities and relations as canonical climate identifiers.The metadata covers standardized model, experiment, variable, institution, and spatial attributes.Source: §3.4 · p. 5

3.5 Hybrid Entity Linking

  • The hybrid linker applies embedding similarity with LLM selection as a fallback, while strict climate-model identifiers use string matching.The automatic linking threshold is τ = 0.80 and the LLM fallback threshold is τf = 0.55.Source: §3.5 · p. 5

4 Experimental Evaluation

The evaluation measures triplet correctness and validation agreement, then reports corpus-scale KG construction, baseline comparisons, knowledge saturation, and external-gold recall.

4.2 Evaluation metric

  • Ctriplet measures the share of generated triplets whose relation is paragraph-supported, allowing implicit references and anaphora without requiring exhaustive gold labels.It is precision-like and emphasizes curator-ready assertions over maximizing coverage.Source: §4.2 · pp. 5–6

4.1 LLM and Human Annotator Agreement

  • Gemini–human Cohen’s κ was 0.640–0.700 versus 0.616–0.689 human–human agreement on 1,000 evidence-support decisions.Against the three annotators’ majority vote, Gemini reached κ = 0.798 and 0.977 observed agreement.Source: §4.1 · p. 5, Table 2 · p. 7

4.3 Results and Discussion

  • Removing the validator from the full pipeline leaves 27,900 triplets at 94.72%; ontology and evidence checks instead provide schema compatibility and traceability.Ontology conformance removes many textually correct but out-of-schema triplets, while evidence validation excludes 845 assertions whose spans cannot be matched to parsed sources.Source: §4.3 · p. 7, Table 4 · p. 8
  • Entity novelty fell from 61.9% to 40.3%, while schema coverage reached 90% of mention types by paper 129 and relation types by paper 434.Of literature-derived edges, 11.0% were independently asserted by at least two papers, which the authors say can partly mitigate per-document recall limits.Source: Figure 3 · p. 6, Table 1 · p. 6, §4.3 · p. 7
  • 23,327 triplets remained after validation at 98.963% correctness; the 2,000-publication KG contains 133,990 nodes and 755,308 relations.The totals distinguish the deterministic initialization from publication-derived additions.Source: §4.3 · p. 7, Table 5 · p. 8
  • On the first 1,007 publications, the system yields 17,141 triplets at 99.067% correctness, versus KARMA’s 3,820 at 92.277%.The comparison adapts KARMA’s prompts to the climate domain.Source: §4.3 · p. 7, Table 6 · p. 8
  • On the 25-paper ClimateIE corpus, the system reaches 92.936% correctness versus 75.004%, while producing fewer triplets.Both systems use the ClimateIE ontology and Gemini-2.5-flash validation.Source: Table 7 · p. 8, §4.3 · p. 8

4.4 Recall and Coverage

  • Only 50.78% of curator-selected gold entities appeared in text; over recoverable entities, pipeline micro-averaged recall was 90.29%.The evaluation used 139 papers with downloadable open-access PDFs and entity metadata.Source: §4.4 · p. 9, Table 8 · p. 9

5 Related work

Prior work frames KG construction as a staged process facing schema, scalability, and factual-reliability challenges. This work combines ontology-constrained extraction, evidence grounding, hybrid entity linking, and validation to build a climate-science KG.

5 Related work

  • KG construction pipelines commonly combine extraction, schema alignment, canonicalization, and validation, while still facing schema conformance, scalability, and factual-reliability challenges.Source: §5 · p. 9
  • Ontology-guided benchmarks assess schema conformance as well as correctness, while SciER and ClimateIE address scientific and climate-specific information extraction, respectively.Source: §5 · p. 9
  • The framework combines ontology-constrained extraction, explicit evidence grounding, hybrid entity linking, and scalable validation to construct a climate-science KG from publications.Source: §5 · p. 9

6 Conclusion

The paper presents an evidence-based, ontology-guided framework for constructing climate-science KGs from publications. Across 2,000 publications, it builds a large KG and reports high correctness after validation, with cleaner assertions than two comparison systems.

6 Conclusion

  • Across 2,000 publications, the pipeline constructs a KG with 133,990 nodes and 755,308 relations while maintaining high triplet correctness after validation.Source: §6 · p. 9
  • The framework combines explicit evidence, ontology checks, and linking to a trusted climate KG backbone to produce high-precision, traceable relations at scale.Source: §6 · p. 9
  • Comparisons with KARMA and ClimateIE show cleaner curator-ready assertions than a general multi-agent scientific IE baseline and a climate-specific IE baseline.Source: §6 · p. 9

Limitations

The pipeline adds verification cost and does not measure triplet-level recall. Its reported measurements do not cover unstated relations or entities outside CMIP metadata families, and LLM validators may misjudge some relations.

Limitations

  • The pipeline's verification stages add about 1% of extraction cost, but the full pipeline costs more than direct extraction and inherits GPU costs for local parsing and screening.Source: Limitations · p. 10
  • Triplet-level recall is not computed because open-ended scientific text would require exhaustive document-level gold triplets.Source: Limitations · p. 10
  • The reported measurements omit relations not explicitly stated in the corpus and triplets whose entities fall outside CMIP metadata families.Source: Limitations · p. 10
  • LLM validators may over-accept plausible but unstated relations or reject relations requiring dispersed evidence or domain knowledge.Source: Limitations · p. 10

Ethical Considerations

The work uses public scientific publications, bibliographic metadata, and community resources, with human review limited to low-risk validation of evidence-supported triplets from non-sensitive text.

Ethical Considerations

  • The work uses publicly available publications, bibliographic metadata, and community resources including ESGF/GCMD, CMIP6 controlled vocabularies, and GeoNames.Source: Ethical Considerations · p. 10
  • Human involvement is limited to low-risk validation of evidence-supported triplets from non-sensitive scientific text.Source: Ethical Considerations · p. 10
  • The authors mitigate risks from third-party LLM extraction and validation through evidence requirements, grounding checks, and attribution and source-license constraints on released resources.Source: Ethical Considerations · p. 10

A Detailed Extraction Methodology

The appendix gives prompt structures and decision logic for agents described in Figures 1 and 7, with the detailed extraction pipeline shown in Figure 5.

A Detailed Extraction Methodology

A.1 Paper Screening Protocols

The screening protocol retains empirical climate research through a strict boolean filter, while publication selection begins from a manual seed and continues through citation mining and LLM filtering.

A.1 Paper Screening Protocols

  • A paper is retained when at least one of three screening criteria receives a positive classification.The criteria concern climate models, observational or simulation datasets, and quantitative experimental results.Source: §A.1 · p. 12
  • The screening agent uses a strict boolean filter to exclude review articles and opinion pieces from the empirical climate research corpus.Source: §A.1 · p. 12
  • The specific screening prompt is provided in Section A.23.2.Source: §A.1 · p. 12
  • The publication-selection pipeline starts with a manual seed, then uses iterative citation mining and LLM-based filtering to build the corpus.Source: Figure 4 · p. 12

A.2 Document level features

Document-level features are obtained from OpenAlex metadata and include both general publication features and domain-specific concepts guided by the target ontology.

A.2 Document level features

  • Document-level features such as title, authors, publication venue, and abstract are obtained from OpenAlex metadata.Source: §A.2 · p. 12
  • The feature set includes general features such as problem, method, and task, which are common to scientific publications.Source: §A.2 · p. 12
  • Ontology-guided domain-specific features capture relevant entities and concepts, including models, datasets, variables, and performance.Source: §A.2 · p. 12

A.3 Graph Initialization Details

This appendix documents the ontology-guided procedures and design choices for extracting, verifying, linking, and initializing climate-science knowledge-graph content, including evidence requirements, conformance checks, and filtering trade-offs.

A.3 Graph Initialization Details

  • Structured ESGF metadata is converted into typed ontology entities and graph edges, with geospatial coverage linked to GeoNames and CMIP6 and GCMD vocabularies enriching model and concept information.Canonical identifiers, such as ESGF variable pr for precipitation, ground later LLM extractions.Source: §A.3 · p. 12

A.4 Entity Linking Categorization Strategy

  • Entity linking routes types to string matching or embedding similarity; rigid model identifiers use string matching to avoid conflating distinct models with high embedding similarity.For sampled candidates, match rates support auto-linking at τ = 0.80 and LLM fallback from τf = 0.55; candidates below τf undergo secondary classification for novel relevant concepts or noise filtering.Source: §A.4 · p. 12, Table 9 · p. 13, §A.5 · p. 13

A.6 Qualifiers and context

  • Qualified claims are represented as paths through first-class Experiment, SubExperiment, Ensemble, Member, Forcing, Metric, and Result nodes rather than as free-text-conditioned edges.The KG contains 35,523 RESULT_HAS_METRIC edges; unsupported qualifiers are removed, and paragraph-level evidence spans retain adjacent context with anaphora resolved.Source: §A.6 · p. 13

A.7 Repeated and cited facts

  • Canonical entity linking consolidates repeated assertions onto shared edges, which retain each asserting paper’s identifier and evidence span; 11.0% of distinct literature-derived edges have at least two paper identifiers.The current provenance design does not distinguish papers that introduced a finding from those that restate it.Source: §A.7 · p. 13

A.8 What triples the pipeline filters remove

  • Within-schema retention was 84.7% (23,085 of 27,262 judge-correct triplets), while Ctriplet rose from 94.05% to 98.96%.The pipeline discarded 14,503 judge-correct triplets: 71.2% for ontology conformance, 5.8% for unlocatable evidence, and 23.0% by the validator agent; human-annotated within-schema retention was 81.9%.Source: §A.8 · p. 13

A.9 Downstream Utility

  • The KG supports multi-hop queries that connect literature-derived assertions with structured climate metadata through auditable paths across models, variables, instruments, methods, and results.The examples include queries about model-produced variables measured by instruments and observational datasets used by methods.Source: §A.9 · pp. 13–14

A.11 Per-Stage Cost and Runtime

  • Verification adds roughly 1% to extraction cost and yields a 4.91-point Ctriplet gain; ontology conformance and evidence validation are deterministic, while each surviving triplet receives one short paid validator call.Extraction costs about USD 200 per 1,000 papers and takes 2–3 hours per 1,000 papers; API cost estimates vary with PDF length and output volume.Source: §A.11 · p. 14

A.12 Entity Classification and Discard Logic

  • Entities that fail string matching and embedding linking below τf receive secondary classification, allowing relevant novel concepts to enrich the graph while filtering noise.Discarded entities are logged to analyze and update exclusion patterns, creating a feedback loop for future classification.Source: §A.12 · p. 14

A.15 Relation Definitions

  • The relation definitions specify a Source’s output frequency, an entity’s spatial resolution, forcings used in an Experiment, Experiment-to-ensemble-member links, regions spanned by an RCM, and subdivision/city-to-Country links.Source: §A.15 · pp. 16–17

A.16 Scientific publications ontology

  • The scientific-publication ontology, adapted from Lan et al. (2025) for climate science, represents papers, methods, innovations, tasks, datasets, metrics, and results.Its relations cover paper-, method-, task-, model-, and result-level links, as well as ontology-aligned extracted relations with paper provenance.Source: §A.16 · p. 17, Entity Definitions · p. 17

A.18 Justification for our selection of climate domain ontology

  • The climate ontology unifies ESGF metadata, GCMD taxonomy, CMIP6 controlled-vocabulary components, and GeoNames identifiers for spatial grounding.These sources provide standardized climate entities and identifiers, model configuration details, and links that support spatially contextualized relations and integration with external knowledge graphs.Source: §A.18 · pp. 17–18

A.19 Ontology Conformance Agent Specification

  • The conformance agent accepts only licensed type-relation tuples and rejects values that violate semantic category guards.The 35-tuple license set enforces relation direction and argument order; guards use CMIP6 vocabulary lists and regular expressions. Decisions are deterministic, auditable, and stored per triple.Source: §A.19 · p. 18

A.20 Per-Stage Examples

  • In pipeline examples, ontology conformance removed a textually correct but unlicensed triple, while 93.4% of triples removed at that stage lacked a schema slot.A flood-impact triple was kept by every stage. An evidence-mismatched claim was dropped, and 79% of validator rejections in the annotated sample were judged valid by the human majority.Source: §A.20 · pp. 18–19

A.21 Evidence Verification Against Parsed PDF and Content and Hallucination Analysis

  • Hallucination rates were 7.74% at an 80% evidence-similarity threshold and 6.14% at 70%, falling to 1.44%, 1.04%, and 0.63% when table and figure text was excluded.The lower rates correspond to thresholds of 80%, 75%, and 70%, respectively.Source: Table 12 · p. 19, §A.21 · p. 19

A.23.1 Annotator Qualifications and Selection

  • Annotators assessed whether supplied evidence explicitly supports each triple and whether it satisfies ontology domain and range constraints, without inferring unstated facts.The three human experts held PhDs in related fields; they received the triple, evidence paragraph, and relevant ontology definitions, and were not asked to perform open-ended extraction.Source: §A.23.2 · pp. 19–20, §A.23.1 · p. 19

Prompts

  • The extraction prompt requests ontology-matched triples with supporting paragraphs and resolved anaphora, while the selection prompt screens titles and abstracts for likely climate-model data or results.The ontology-conformance and evidence-validator agents use no prompt because they are deterministic. AI assistants were used only for manuscript wording and pipeline-figure support, not scientific decisions or experimental results.Source: §A.22 · p. 19, Prompts · p. 25
  • Entity-linking fallback selects exactly one listed instance or no match, and validation accepts explicitly stated or clearly implied relations while allowing contextual resolution of implicit references.The validator may use domain knowledge for abbreviations and account for anaphora; unsupported or contradicted relations are invalid.Source: Prompts · pp. 26, 28
  • Prompts extract research field, keywords, problem, method, task, and contribution from native-format PDFs in short, prescribed formats.The prompt examples specify one phrase for field, 3–5 keywords, and one sentence of at most 50 words for problem, method, task, and contribution.Source: Prompts · pp. 26–27
  • A screening prompt requires both an observational dataset and a quantitative evaluation metric, while table extraction returns model, dataset, metric, and result values using table content only.The table prompt requests quantitative values and directs the extractor to use “no” when a required field is missing.Source: Prompts · pp. 27–28
Loading 2610.32725v1…