PaperlayerView exact revision on arXiv

Source-linked AI summary

Automatically Building and Updating a Knowledge Graph of MLIP Models

Alexis Beer, Liudmyla Klochko, Mathieu d'Aquin

arXiv:2610.09644v1cs.AI

TL;DR

Rapidly appearing MLIP models are difficult to track, including how they differ, were trained, and fit particular applications. The paper proposes ontology-guided LLM extraction with SHACL validation and correction; applied to Matbench Discovery models, it produced a graph of 49 variants from 31 families and reported release recency as a better performance indicator than model size.

  • Problem

    Rapidly emerging MLIP models make it harder to track proposals, distinguish them, and assess their training and suitability for particular applications.

    Source: §1 · p. 1

  • Method

    The paper uses an ontology-guided LLM process to extract model information and an LLM-SHACL loop to detect and correct structural inconsistencies.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    49 model variants from 31 families populate the graph, and release recency is reported as a better performance indicator than model size.

    Source: §7 · pp. 7, 9

  • Takeaways & Limitations

    The graph supports structured queries about model datasets, architectures, and relationships between model size and predictive performance.

    Source: §7 · p. 7

    SHACL conformance ensures consistency with defined constraints, but cannot guarantee that extracted information is factually correct.

    Source: §8 · p. 11

Abstract

from arXiv · show

Complementing the many efforts in providing semantic representations of concepts, notions, and entities in materials science, we report and illustrate a process by which we can automatically build a knowledge graph of the fast evolving field of machine learning applied to the prediction of material properties, focusing on MLIP (Machine Learning Interatomic Potential). This LLM-based process relies on multiple steps, from information extraction in documents and articles to a validation loop using SHACL constraints to detect and correct errors. It is carried out on a model-by-model basis, focusing on the consistency of representation, therefore enabling an iterative construction where the addition of new models is facilitated. We illustrate the process by showing a few interesting aspects that can be queried from a knowledge graph built from the models listed in the Matbench Discovery leaderboard.

1. Introduction

The rapid evolution of MLIP models makes it harder to track new proposals, distinguish them, and assess their training and suitability for particular applications. The paper proposes an ontology-guided, LLM-based process for organizing scattered model information in a continuously updated knowledge graph.

1. Introduction

  • The fast-growing MLIP field makes it difficult to track new models, distinguish them, and judge their training and suitability for particular applications.The paper says this requires continuous monitoring of leaderboards, benchmarks, and scientific literature.Source: §1 · p. 1
  • The proposed knowledge graph organizes scattered information about MLIP models to support ongoing monitoring.The goal is detailed descriptions of models and their training and evaluation.Source: §1 · p. 1
  • An LLM-based extraction and construction process, guided by a purpose-built ontology, populates the graph from articles and other textual resources.The process also uses an LLM to correct errors identified by SHACL constraint tests.Source: §1 · p. 2
  • The paper presents the work as a response to rapid development that makes manual knowledge-graph population insufficient, and outlines its ontology, extraction, correction, and leaderboard-graph components.It reports that code, ontologies, and results are publicly available.Source: §1 · p. 2

2. Related work

Prior work covers semantic representation in materials science and ontologies for machine-learning models, but the authors focus on representing MLIP models for materials. Their proof of concept uses a purpose-built ontology and textual resources associated with Matbench Discovery.

2. Related work

  • Materials-science knowledge graphs and ontologies have supported semantic interoperability and integration of heterogeneous data sources, while prior NLP work includes materials extraction pipelines and LLM-based knowledge-graph construction.Examples include mat2vec, materials-focused NLP, and automatic knowledge-graph extraction.Source: §2 · p. 2
  • ML Schema [14] provides classes and properties for describing machine-learning experiments and relies on other ontologies for high-level model, implementation, and evaluation descriptions; SML describes models, datasets, and evaluation.SML focuses more on usability for practical applications than on experiments.Source: §2 · p. 2
  • For Matbench Discovery, the textual resources used to describe models include PDF articles and YAML files with basic information and dataset links.These are the resources referenced in the paper’s account of model information sources.Source: §2 · p. 2
  • The paper targets granular MLIP descriptions—including architectures, training parameters, versions, variants, evaluations, and prediction results—to support uses such as model recommendation.Its proof of concept uses a purpose-built ontology, with refinement and alignment to existing ontologies left for future work.Source: §2 · p. 3

3. Overview of the approach

The process extracts model information from textual resources into an ontology-structured knowledge graph, then validates and corrects it with SHACL constraints and an LLM. This model-by-model workflow is intended to support consistent integration and updates as new models appear.

3. Overview of the approach

  • For each model, an LLM extracts information from textual resources with the ontology guiding both the questions asked and the graph structure produced.A separate conversion step turns extracted JSON into RDF using relevant ontology classes and properties.Source: §3 · p. 3
  • SHACL Shapes [3] act as integrity constraints; when validation finds inconsistencies, an LLM uses the original resources, graph, and validation report to correct them in a loop.The loop ends when detected inconsistencies have been removed.Source: §3 · p. 3
  • Ontology conformance and SHACL validation support integrating each model graph into a global graph that can be updated when new models appear.Passing the explicitly defined constraints does not establish completeness or factual correctness.Source: §3 · p. 3

4. An Ontology of ML Models

The ontology represents MLIP variants, their architectures, training runs, and evaluation results in a modular structure. It accommodates model families and variants, training strategies, benchmark metrics, and predictions on individual inputs.

4. An Ontology of ML Models

  • The ontology separates model architecture, training, and evaluation into three modules, with evaluation covering both benchmark metrics and individual-input results.Its requirements include model versions and families, architecture, training data, and evaluation.Source: §4 · p. 4

4.1. The core ML Model Ontology

  • ModelVariant identifies the specific model version, while ModelFamily groups related variants and the architecture can represent nested components and configurations.Architecture descriptions can include component roles such as backbone or head, as well as parameter counts or layer counts.Source: §4.1 · p. 4

4.2. The ML Training Module

  • MachineLearningTrainingRun connects a model variant to its training setup, including datasets, loss functions, optimizers, and hyperparameters.The module also represents strategies such as distillation and fine-tuning that use existing model materializations.Source: §4.2 · pp. 4–5

4.3. The ML Evaluation Module

  • MachineLearningEvaluationRun represents a variant evaluated on a task and produces a BenchmarkResult grouping MetricResults.Individual-input predictions store input identification, predicted values, and expected values, allowing metrics to be recalculated on dataset subsets.Source: §4.3 · p. 5

5. LLM-based Process for Extracting Model Details from Documents

The process extracts model details from textual sources, first into a question-guided JSON structure and then into RDF aligned with the ontology. It records evidence and provenance for extracted information, allowing queries about both models and information origins.

5. LLM-based Process for Extracting Model Details from Documents

  • For each model variant, the process uses its reference PDF and Matbench YAML description, while individual predictions are handled separately by script.The PDF and YAML collectively provide details on the model, training, and evaluation.Source: §5 · pp. 5–6
  • A JSON map links ontology properties to English questions, guiding the LLM to extract values into JSON before a separate step converts them into ontology-aligned RDF.The RDF-generation step receives both the extracted JSON and relevant ontology classes and properties.Source: §5 · p. 6
  • The LLM provides a document reference or quotation as evidence for each extracted value, and RDF-star records that source as an annotation on the corresponding triple.The paper illustrates this with a layer-count value of 7 traced to Appendix Table 10 for MPtrj training.Source: §5 · p. 6
  • The resulting graph can be queried for detailed model information and the provenance of that information.The example represents the architecture configuration's layer count as 7 and identifies its source.Source: §5 · pp. 6–7

6. SHACL-based Validation and Correction of the Extracted Knowledge Graph

The process validates each model's knowledge graph against SHACL constraints and uses reported violations to guide an LLM correction loop. Passing those constraints does not guarantee that extracted information is factually correct.

6. SHACL-based Validation and Correction of the Extracted Knowledge Graph

  • A graph that passes SHACL constraints is structurally consistent but is not thereby guaranteed to contain factually correct information.Checking every generated triple manually would undermine the aim of automating graph construction.Source: §6 · p. 7
  • SHACL constraints check graph structure, completeness, and integrity, and a second LLM uses the validation report and original sources to correct reported issues.The process repeats until SHACL identifies no new issues, yielding a graph valid against the established constraints.Source: §6 · p. 7
  • For AlchemBERT, a missing pre-training dataset violated a minimum-count constraint, and the repair LLM added the missing dataset relation.Subsequent SHACL validation confirmed that this violation was resolved.Source: §6 · p. 7

7. Results and Discussion

The knowledge graph is used to compare MLIP performance across release date, size, architecture, and training datasets. The reported patterns associate better performance with recency and, in dataset comparisons, with multi-dataset training.

7. Results and Discussion

  • Larger models generally obtain better performance, with pet-oam-xl-1.0.0 the largest and lowest-MAE model and cgcnn among the smallest and highest-MAE.Alchembert is a notable outlier: relatively large but relatively poor-performing.Source: §7 · pp. 7–8
  • Recency tends to indicate performance better than model size, although newer models also tend to be larger.The comparison uses the date each model was added to the Matbench Discovery leaderboard.Source: Figure 5 · p. 8, §7 · pp. 8–9
  • Equivariant message-passing architectures and graph transformers perform consistently well on the convex hull distance regression task, while pure transformers are not competitive.General message-passing and GNN architectures can also perform well; graph convolution was not pursued further after early attempts.Source: Figure 6 · p. 9, §7 · p. 9
  • Across non-convolution graph-based methods, models have achieved MAEs at or below 0.02, suggesting size or training data may matter more than architecture.The broad GNN category spans models that could not be assigned to a more specific architecture, helping explain its wide performance range.Source: §7 · p. 10
  • Models trained with OMat24 and sAlex achieve significantly better average MAE than MPtrj-only models, but the pattern may reflect multi-dataset training rather than dataset superiority.Models trained on multiple datasets appear in each corresponding group, and many sAlex models also used OMat24 and MPtrj.Source: Figure 7 · p. 10, §7 · p. 10

8. Conclusion

The paper presents an automatic, incremental process for building a knowledge graph of MLIP models in the Matbench Discovery leaderboard, with evidence recording and an LLM/SHACL validation loop. It also identifies factual validation and model recommendations as areas for further work.

8. Conclusion

  • The process builds a knowledge graph of Matbench Discovery MLIP models, recording evidence for extracted information and using an LLM/SHACL loop to keep model, training, and evaluation descriptions structurally consistent.The graph was queried to create visualizations of model characteristics and performance.Source: §8 · pp. 10–11
  • SHACL validation checks structural consistency but cannot guarantee that extracted information is factually correct.The authors say gold-standard evaluation is infeasible because constructing a model’s graph description is excessively complex and time-consuming.Source: §8 · p. 11
  • Evidence attached to extracted triples can support manual validation, but reviewing it remains complicated and time-consuming.Source: §8 · p. 11
  • A proposed next step is a recommendation engine that relates model performance to material characteristics and helps identify suitable models for specific tasks and material sets.The authors note that aggregate metrics such as MAE may not reflect differences across subsets of tested materials.Source: §8 · p. 11

Declaration of use of Generative AI

Generative AI was used in the paper’s information-extraction and correction process, in development activities, and for grammar and style improvements to the manuscript.

Declaration of use of Generative AI

Loading 2610.09644v1…