PaperlayerView exact revision on arXiv

Source-linked AI summary

Extracting ontology-compliant knowledge from scientific text describing irradiated materials using large language models

Marco Luca Sbodio, Marcos Martínez Galindo, Vanessa Lopez, Blanca Biel, Pablo Canca, Pedro Delgado, Jesús I. Mendieta-Moreno, Raphael Tack, Maria J. Caturla

arXiv:2609.17291v1cs.AI

TL;DR

Critical information for modeling fusion-reactor materials is embedded in unstructured scientific literature, limiting structured reuse and making manual extraction time-consuming. eolas combines a domain ontology, large language models, configurable extraction components, and a validation interface to produce ontology-compliant document-level knowledge graphs. Few-shot prompting achieved 67.6% success at NGED*=0.2, while zero-shot prompting achieved 36.0%, and extensive experiments provided practical extraction guidelines.

  • Problem

    Critical information for modeling fusion-reactor materials is embedded in unstructured scientific literature, limiting structured reuse and making manual extraction time-consuming.

  • Method

    eolas combines a domain ontology, large language models, configurable extraction components, and a validation interface to produce ontology-compliant document-level knowledge graphs.

  • Results

    Few-shot prompting achieved 67.6% success at NGED*=0.2, while zero-shot prompting achieved 36.0%, and extensive experiments provided practical extraction guidelines.

  • Takeaways & Limitations

    The extracted knowledge graphs can provide parameters for atomistic simulations and support high-throughput studies while reducing manual setup effort and error.

  • Takeaways & Limitations

    The study focuses on irradiated materials, uses strict equality in graph-edit-distance evaluation, and identifies graph merging across paragraphs or documents as an unresolved challenge.

Abstract

from arXiv · show

The quest for new materials increasingly relies on predictive models and comprehensive simulations that span scales from atomic to macroscopic levels. However, essential data necessary for these models and simulations are often embedded in scientific literature as unstructured text, limiting reusability and posing challenges for researchers seeking to leverage existing knowledge effectively. While extracting structured data from unstructured text using large language models is gaining popularity, traditional methods typically generate key-value pairs data with straightforward schemas. In contrast, we introduce eolas, a modular pipeline that uses large language models to automatically transform scientific documents into knowledge graphs aligned with a specified ontology. We demonstrate eolas effectiveness in extracting useful information for scientists studying materials designed to endure the extreme temperatures and radiation levels found in fusion reactors. While a human expert might spend between thirty to ninety minutes extracting relevant data from an article, eolas can generate high-quality knowledge graphs in just a few minutes. These are presented in a tabular format with faceted navigation for easy human validation. Additionally, we introduce the first benchmark dataset designed to assess large language models capabilities in constructing knowledge graphs within the domain of irradiated materials. The analysis of 168 experiments using our dataset, various large language models and prompting techniques provides key insights that we summarize into practical guidelines for effectively extracting knowledge graphs aligned with an input ontology.

1 Introduction

The paper addresses the difficulty of structuring complex irradiated-materials knowledge embedded in scientific literature by combining a domain ontology with large language models. It introduces eolas, a modular pipeline that extracts ontology-compliant knowledge graphs for inspection, validation, and reuse.

  • Motivation: Irradiated materials contain complex defect types and relationships, including vacancies, interstitials, Frenkel pairs, dumbbells, clusters, and grain boundaries.The ontology targets semantic relationships within defect energetics and atomic-scale defect modeling.
  • Motivation: Structured data for irradiated-materials modeling is scarce because critical information remains embedded in dense scientific literature.This limits consolidation of knowledge for predictive models and makes manual extraction time-consuming.
  • Approach: The study develops an ontology-informed approach that uses LLMs to extract machine-readable information about atomistic defect modeling with minimal human intervention.The ontology was derived from domain literature and scientist interviews and uses RDF and OWL standards.
  • Approach: eolas transforms text into document-level knowledge graphs aligned with an input ontology and presents them as queryable, serializable, or tabular data linked to source passages.The interface supports expert inspection and validation of extracted knowledge.
  • Evaluation: The work introduces the first benchmark dataset for ontology-compliant knowledge-graph extraction and evaluates models, prompts, schema representations, and post-processing across 168 experiments.Few-shot examples pair passages with manually reviewed ontology-compliant graphs for in-context learning.
  • Scope: The ontology and benchmark are open source, while the ontology, dataset, and pipeline are presented as adaptable beyond irradiated materials.The broader applicability is attributed to their focus on atomic-scale defects and domain-agnostic extraction.

2 Results

Across 168 experiments, few-shot prompting generally outperformed zero-shot extraction, while serialization, instruction detail, and consolidation affected performance under specific conditions. The study also produced a benchmark and practical guidelines for ontology-compliant knowledge-graph extraction.

  • 2 Results: The benchmark contains 126 passage–knowledge-graph pairs, with 111 passages reserved for evaluation, and the ontology comprises 14 classes, 111 instances, 13 object properties, and 25 data properties.The ontology and benchmark dataset were publicly released, and the experiments evaluated six openly accessible LLMs.
  • 2 Results: 45.9% few-shot success at NGED*=0.1 exceeded the best zero-shot result of 15.3% by 200%.The best zero-shot configuration used LLAMA_4, detailed instructions, TTL serialization, instances, and consolidation; the best few-shot configuration used the same setup with LLAMA_3.
  • 2 Results: At NGED*=0.2, the best zero-shot result was 36.0%, while few-shot configurations reached 67.6% with GPT_120 and 66.7% with GPT_20.BAML serialization appeared more effective than TTL at this higher error tolerance, particularly in the reported few-shot configurations.
  • 2 Results: Few-shot configurations significantly outperformed zero-shot configurations across all NGED* thresholds, with success increasing as error tolerance rose.The zero-shot family also showed a wider interquartile range as NGED* increased, indicating greater variability than few-shot configurations.
  • 2 Results: Prompt features had conditional effects: instances usually did not significantly change performance, while consolidation mainly helped zero-shot settings at strict thresholds.Instruction and serialization differences were not significant at NGED*=0.1, but at higher thresholds detailed instructions improved TTL few-shot configurations and BAML outperformed TTL in zero-shot settings.
  • Guidelines: The resulting guidelines favor validated few-shot examples, detailed instructions with TTL, BAML with simple instructions in zero-shot settings, and ontology-guided consolidation for smaller or zero-shot models.The authors report that high-quality examples provide stronger inductive signals than instance lists and that in-context learning supports better semantic correctness than post-processing alone.

3 Discussion

The discussion highlights eolas’s practical value for extracting ontology-compliant knowledge graphs from irradiated-materials literature and supporting downstream scientific workflows. It also identifies evaluation and graph-merging limitations that motivate future work.

  • The study evaluates ontology-compliant extraction through 168 experiments spanning multiple LLMs, prompting techniques, ontology representations, and post-processing configurations.The experiments produced evaluations of 18,648 knowledge graphs and systematically varied six LLMs, instruction formats, serialization formats, instances, examples, and consolidation heuristics.
  • The evaluation metric strictly penalizes synonyms, approximations, and correct information added beyond the benchmark graph, motivating approximate matching and heuristic extensions.The authors note that this strict normalized graph edit distance can understate performance when extracted content is semantically valid but not textually identical to the benchmark.
  • Despite immediate applications, merging paragraph-level graphs into coherent document- or cross-document graphs remains an unresolved challenge.The authors identify interactive LLM-based multi-agent systems with human interaction as one possible future direction.
  • Eolas’s extracted knowledge graphs provide structured parameters for atomistic simulations and can reduce manual setup errors in high-throughput studies.The cited parameters include migration energies, binding energies, and defect configurations.
  • Ontology-based knowledge graphs can improve domain-specific question answering when integrated into Graph-RAG systems.A simplified Graph-RAG experiment using the ontology and expert-defined competency questions confirmed improved answer accuracy.

4.1 Ontology/Schema building

The ontology represents irradiated-materials knowledge through expert-defined concepts, properties, instances, and semantic constraints. Its structure was refined to capture contextual information needed for simulations and experiments and to support tabular presentation.

  • 4.1 Ontology/Schema building: The ontology captures concepts and semantic relationships for atomistic defect energetics in irradiated materials, covering simulation and experimental studies.
  • 4.1 Ontology/Schema building: Domain experts identified representative examples whose properties and values informed ontology classes, properties, instances, and semantic constraints.
  • 4.1 Ontology/Schema building: Leaf properties become table columns and leaf nodes provide values, while connected intermediate-node values remain in the same row.
  • 4.1 Ontology/Schema building: Expert-defined competency questions were used to refine whether the ontology captured contextual information required for ground-truth construction.
  • 4.1 Ontology/Schema building: Figure 5 presents the proposed ontology structure for modeling irradiated materials.

4.2 eolas extraction pipeline

eolas is a modular, ontology-guided pipeline that parses documents, extracts fragment-level graphs with configurable LLM prompts, and consolidates them into document-level knowledge graphs. Optional annotation, filtering, serialization, and hallucination-handling components support efficiency, schema alignment, and expert validation.

  • 4.2 eolas extraction pipeline: eolas preprocesses PDFs, extracts ontology-guided graphs from text fragments, and consolidates partial graphs into document-level knowledge graphs.
  • 4.2 eolas extraction pipeline: Optional annotators and schema-driven filtering remove irrelevant passages, improve entity coverage, support hallucination detection, and reduce LLM processing costs.
  • 4.2 eolas extraction pipeline: The extractor configuration controls instruction detail, ontology serialization format, and inclusion of instances or examples in LLM prompts.
  • 4.2 eolas extraction pipeline: The pipeline provides TTL, VERB, and BAML extractor implementations, with BAML converting ontology structures into classes, fields, and enumerations for JSON-aligned output.
  • 4.2 eolas extraction pipeline: Prompt size depends on fragment length, ontology size, serialization format, and whether instances or examples are included.
  • 4.2 eolas extraction pipeline: Hallucination heuristics exclude datatype values lacking textual matches and relations or classes outside the ontology, prioritizing precision over recall.
  • 4.2 eolas extraction pipeline: Current consolidation heuristics interconnect entities across a document, while advanced subgraph merging across paragraphs remains future work.

4.3 eolas user interface

The eolas interface presents document-level and fragment-level knowledge graphs as structured tables. Faceted filters and text highlighting help users focus on relevant properties, values, and source passages during exploration and validation.

  • 4.3 eolas user interface: The interface represents document-level graph instances as rows and datatype properties as columns in a tabular visualization.
  • 4.3 eolas user interface: Inclusion and exclusion filters address sparse tables with many ontology-wide columns and enable faceted navigation by selected datatype properties.
  • 4.3 eolas user interface: Users can filter further by selecting particular datatype-property values to narrow the displayed information.
  • 4.3 eolas user interface: Fragment-level views show subgraphs in tables and highlight relevant source text when annotations are available.
  • 4.3 eolas user interface: Knowledge graphs can be visualized at both document-level and fragment-level granularity.

Appendix A Additional results: percentage of success at different thresholds

The appendix evaluates 168 experimental configurations using percentage success at three normalized graph-edit-distance thresholds. Tables 4–6 report results for NGED*=0.05, 0.15, and 0.25, respectively.

  • Appendix A Additional results: percentage of success at different thresholds: 168 configurations are evaluated using percentage success at normalized graph edit-distance thresholds NGED*=0.05, NGED*=0.15, and NGED*=0.25.
  • Appendix A Additional results: percentage of success at different thresholds: Table 4 reports Pj(0.05) as a percentage for every experimental configuration.
  • Appendix A Additional results: percentage of success at different thresholds: Table 5 reports Pj(0.15) as a percentage for every experimental configuration.
  • Appendix A Additional results: percentage of success at different thresholds: Table 6 reports Pj(0.25) as a percentage for every experimental configuration.

Appendix B Competency Questions and Graph-RAG

Ontology-compliant knowledge graphs normalize scientific information for uniform querying and can support domain-specific multi-hop questions through Graph-RAG. The appendix illustrates both the graph construction context and the ability of ground-truth and predicted graphs to preserve query-relevant relationships.

  • Appendix B Competency Questions and Graph-RAG: Ontologies normalize terminology and encode semantic relationships, supporting cross-document queries, logical inference, and detection of implausible outputs.Competency questions evaluate whether an ontology represents the concepts, relationships, and contextual information needed by its intended application.
  • Appendix B Competency Questions and Graph-RAG: Ontology-compliant KGs can be rendered as tables, queried with SPARQL, or integrated into Graph-RAG pipelines.They retain contextual information from document-level extraction while supporting uniform access.
  • Appendix B Competency Questions and Graph-RAG: Figure 7 shows that standard RAG failed on a formation-energy question, whereas naive Graph-RAG retrieved the correct answer from ground-truth or predicted KGs.The question required associating multiple formation-energy values with their distinct methods and reference sources.
  • Appendix B Competency Questions and Graph-RAG: Ground-truth KGs from 126 curated passages supported competency questions spanning multiple paragraphs or documents.The complete ground-truth KG was small enough to fit in a large LLM context window, so retrieval was unnecessary for this evaluation.

Appendix C Effect of prompting strategies

The prompting study examines how ontology representations, instructions, and few-shot examples shape knowledge-graph extraction. It finds that few-shot prompting and detailed instructions improve alignment and completeness, while hallucinations, omissions, constraint violations, and evaluation sensitivity remain concerns.

  • Appendix C Effect of prompting strategies: Few-shot prompting significantly improves semantic alignment and structural correctness, especially for large passages containing entangled multi-entity relationships.Zero-shot errors often involve formatting conventions and confusion between closely related properties.
  • Appendix C Effect of prompting strategies: Hallucinations occur across models, particularly with zero-shot prompts and passages lacking sufficient data.Models may add domain-reasonable triples that are absent from the text and ground truth.
  • Appendix C Effect of prompting strategies: Models also omit relevant information explicitly present in the passage, producing partially accurate but incomplete triple sets.This omission is a distinct failure mode from adding unsupported background knowledge.
  • Appendix C Effect of prompting strategies: Detailed instructions direct models to follow ontology constraints, include stated entities, avoid hallucinations, and respect domain, range, and functional-property restrictions.Models still occasionally violate constraints, such as representing multiple defect types without creating separate nodes.
  • Appendix C Effect of prompting strategies: The evaluation metric penalizes notation and formatting deviations even when experts might judge the extracted values valid.This makes measured graph-edit differences sensitive to representational variation, not only substantive extraction errors.
  • Appendix C Effect of prompting strategies: Few-shot examples were manually selected for property coverage, but their model-specific selection was not evaluated.The authors suggest that tailoring examples to each model and adding more examples as token limits ease could improve results.
Loading 2609.17291v1…