PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Materials Data and Knowledge Extraction using Machine Learning and Large Language Models

Aravindan Kamatchi Sundaram, B. Pabitramohan Prusty, Rohit Batra

arXiv:2610.29895v1cond-mat.mtrl-scics.AI

TL;DR

Materials evidence is abundant but remains scattered across heterogeneous literature rather than machine-actionable form. This review surveys extraction approaches and identifies incomplete sample-level records and limited standardized benchmarks among the field's persistent challenges.

  • Problem

    Materials findings are largely embedded in unstructured or semi-structured literature, while only a small fraction is available in structured form for systematic use.

    Source: §1 · p. 1

  • Method

    The review traces materials extraction approaches and examines their data resources, accuracy, costs, modalities, opportunities, and limitations.

    Source: §1 · pp. 2–3

  • Results

    The review identifies incomplete sample-level records, limited automation for synthesis and figures, and a shortage of standardized benchmarks as persistent gaps.

    Source: §2 · p. 4, §6 · p. 25

  • Takeaways & Limitations

    Explicit roles and well-defined extraction tasks are practical prompt-design rules, though model capability can yield larger gains than prompt or pipeline changes.

    Source: §5.1.3 · p. 15

    Current databases often omit parts of the sample-level structure-processing-property-performance record, while small task-specific benchmarks hinder comparisons.

    Source: §6 · p. 25

Abstract

from arXiv · show

Over the past decade, substantial efforts have been devoted to developing machine learning (ML) methods for extracting materials data and, where possible, materials knowledge directly from the scientific literature. This review traces the evolution of these methods from heuristic, rule-based pipelines through natural language processing (NLP) models and transformer-based encoders to, lastly, large language models (LLMs) and agentic systems capable of extracting materials information across multiple modalities, including text, tables, and figures. Beside construction of large, queryable materials databases and knowledge graphs that support downstream materials discovery, we discuss how these advances have enabled new capabilities, such as, generic question answering across materials science, AI co-scientists and chatbots for research assistance, and closed-loop optimization and discovery when integrated with autonomous laboratories. We critically assess the datasets generated using these approaches, their extraction accuracy and computational costs, and the respective strengths and limitations of different methodologies, and even provide practical guidelines for developing more reliable and scalable materials data extraction pipelines. Finally, we identify key open challenges, including the extraction of full materials structure-property-processing-performance tetrahedron at the sample level, the scarcity of reliable benchmark datasets for systematic evaluation, and the resolution of non-standard materials terminology across different document modalities.

1 Introduction

Materials findings are abundant but remain dispersed across unstructured and semi-structured literature, limiting systematic access at scale. This review traces extraction approaches and their challenges, from curated and computational databases to NLP, transformers, and LLMs.

1 Introduction

  • Materials data are individual observations, whereas materials knowledge compresses observations into relationships, trends, rules, and models that support prediction.Researchers form such knowledge by comparing findings across studies, but their ability to integrate connections across subfields is limited.Source: §1 · pp. 1–2
  • The central bottleneck is converting information already present in diverse publications into reliable, machine-actionable records that manual curation cannot produce at literature scale.The literature spans text, tables, figures, captions, and supporting files, while manual curation remains limited to particular materials classes and use cases.Source: §1 · pp. 1–2
  • Expert-curated repositories are reliable but slow to scale, while high-throughput computational databases cover many structures but miss experimental synthesis, processing, disorder, dopants, and device behavior.The review names Landolt-Börnstein, ICSD, the Pauling File, and PoLyInfo as curated resources, and Materials Project, AFLOW, OQMD, NOMAD, and Alexandria as computational platforms.Source: §1 · p. 2
  • Extraction methods progressed from hand-crafted NLP rules to contextual transformer encoders and then LLMs and agentic systems that target broader records and multiple modalities.The review assesses these approaches' opportunities and limitations, along with extraction accuracy, cost, and open challenges.Source: §1 · pp. 2–3

2 The forms of materials data and the difficulty of extracting them

Materials data types differ substantially in extraction maturity and ease of automation. Structured crystallographic files are comparatively tractable, while property, synthesis, image, and spectral information face distinct interpretation and linking challenges.

2 The forms of materials data and the difficulty of extracting them

  • Crystallographic data are highly mature and automatable because standardized CIF files encode well-defined fields and can be parsed directly.Dedicated repositories have accumulated on the order of 10^5–10^6 entries.Source: §2 · p. 3
  • Property extraction is moderately mature but challenged by variable terminology, document-spanning dependencies, difficult table layouts, and implicit meanings of reported values.A reported composition may be a nominal synthesis target rather than an experimentally measured composition, and property conditions can be misinterpreted.Source: §2 · pp. 3–4
  • Synthesis and processing extraction remains low in maturity and automation because records require linking sequential operations, quantities, and conditions that may be implicit or inherited.Unlike crystallographic files, synthesis reporting lacks a comparable structured standard.Source: §2 · p. 4
  • Characterization images are least developed, while diffraction and other spectra vary in maturity but remain difficult to automate when information must be recovered from figures.Image interpretation requires visual and contextual reasoning; figure-based spectra likewise require multimodal interpretation of plots and surrounding text.Source: §2 · p. 4

3 High level Pipeline

Materials extraction pipelines collect and clean literature, extract entities and relations, reconcile results, and store structured records. Their design choices trade off flexibility, completeness, accuracy, and computational cost.

3 High level Pipeline

  • The general pipeline moves from document acquisition and cleaning through extraction and reconciliation to storage as a queryable database or knowledge graph.The overview presents this skeleton as common to both hand-written rules and language-model workflows.Source: §3 · p. 5, Figure 2 · p. 5

3.1 Procuring and pre-processing the content

  • Document procurement and preprocessing preserve structure, repair formula and unit artifacts, and chunk content; errors such as truncation or garbled formulas propagate downstream.Structured XML is preferred when available, while PDFs may require layout-aware parsing.Source: §3.1 · p. 5

3.2 Extraction workflows

  • Rule-based pipelines offer traceable precision but brittle rules; transformers learn contextual relations but need labeled task training; LLMs are flexible yet risk hallucination, prompt sensitivity, and high corpus-scale cost.The approaches also differ in context handling and task specialization; multi-agent workflows can mitigate LLM costs to some extent.Source: §3.2 · pp. 6–7

3.3 The target data schema

  • A target schema defines entity and relation types; transformer schemas are fixed during training, whereas LLM schemas can be changed at inference time, trading consistency for flexibility.Generic schemas support broad databases but reduce accuracy relative to domain-specific schemas because property identification is left to the LLM.Source: §3.3 · p. 7

3.4 Reconciliation and post-processing

  • Reconciliation reconnects facts scattered across a document, while normalization canonicalizes identifiers and units so records can be compared, deduplicated, and queried.This stage turns isolated extractions into coherent records, but interdependency resolution remains underdeveloped and difficult.Source: §3.4 · p. 7

3.5 Extraction accuracy

  • High-80s to mid-90s precision and recall are typical for state-of-the-art LLM pipelines, but human verification remains necessary and bespoke evaluation sets hinder comparisons.Materials extraction lacks broad standardized benchmarks, so studies commonly rely on smaller expert-annotated datasets.Source: §3.5 · pp. 7–8

4 ML-based extraction approaches

This section reviews materials-data extraction from rule-based NLP and static embeddings through transformers and BERT, covering database construction, methods, and limitations. The approaches support literature-scale extraction, but challenges remain in relation extraction, annotation costs, and linking complete materials records.

4.1 Pre-transformer approaches: Static word embeddings and rule-based parsing (2015–2021)

  • Rule-based pipelines built large materials databases, but fixed grammars achieved high precision mainly for anticipated patterns and typically suffered from low recall.Examples span battery, thermoelectric, perovskite, MOF, and zeolite data; applying organic-material rules to inorganic literature reduced precision from 90% to 66%.Source: §4.1 · p. 8
  • Materials-trained embeddings supported NER and entity normalization at scale, while unsupervised vectors also surfaced material–property associations and candidate thermoelectrics.A BiLSTM-CRF model achieved 0.87 overall F1 and was applied to 3.27 million abstracts, extracting over 80 million entities for Matscholar; mat2vec used roughly 3.3 million abstracts without labels.Source: §4.1 · pp. 8–9

4.2 The transformer era: Contextual embeddings, foundation models, prompting and fine-tuning

  • Transformers provide context-dependent token representations and can be pretrained on unlabelled corpora, then adapted through fine-tuning or prompting for different applications.The review distinguishes encoder-only, encoder–decoder, and decoder-only models, and uses “LLM” for models with more than roughly a billion parameters.Source: §4.2 · p. 9

4.3 Scientific BERT models

  • Domain-specific scientific encoders consistently outperformed general-purpose BERT on materials benchmarks, with domain-specific pretraining improving NER by 1–12%.The reported gains were larger in data-scarce regimes, while a 770M-parameter ScholarBERT offered little advantage over substantially smaller domain-specific models.Source: §4.3 · p. 10

4.3.1 Fine-tuning BERT models

  • BERT-based relation extraction uses pair classification, joint entity–relation models, question answering, or heuristic links, but relation F1 can trail entity F1.DyGIE++ scored 0.82 F1 for entities and 0.66 for relations; SciBERT pretraining required about one week on an 8-core TPU v3, while extending tasks often shifts the dominant cost to expert annotation.Source: §4.3.1 · p. 10

4.3.2 Databases extracted using BERT models

  • MatKG scaled co-occurrence graph construction to about 5 million papers, yielding 85 million entity mentions, about 70,000 unique entities, and 5.4 million weighted edges.Its edges indicate co-occurrence, not a specific scientific relation.Source: §4.3.2 · pp. 10–11
  • Explicit-relation databases span polymer and battery applications, but extraction performance declines as tuples grow: F1 falls from 0.69 for two-entity tuples to 0.58 for three and 0.43 for full quaternary tuples.PolyIE formalized full-text polymer NER and relation extraction; BatteryBERT also used question answering for device-component assignments.Source: §4.3.2 · p. 11
  • BERT-based resources also capture synthesis and specialized property records, but the reviewed databases do not comprehensively link composition, structure, processing, and properties in one entry.They mainly record individual relations, such as material–property or material–process links.Source: §4.3.2 · p. 11

5 Large language models based extraction approaches

BERT-based extraction requires dedicated annotation and is constrained by short context windows and text-only inputs. General-purpose LLMs can use prompting across tasks and larger contexts, while multimodal models can extract information from figures as well as text and tables.

5 Large language models based extraction approaches

  • LLMs can process larger portions of papers jointly than BERT’s typical 512-token window, potentially retaining links between information reported in different sections or supplementary materials.The cited passage describes modern LLM context windows extending to hundreds of thousands or millions of tokens.Source: §5 · p. 11
  • Prompting a frozen general-purpose LLM can avoid task-specific annotations for end users and support varied extraction tasks by changing the prompt.The review describes this as shifting effort from model training toward prompt and workflow design.Source: §5 · pp. 11–12
  • Multimodal LLMs can extract information from figures, including data reported in diffraction patterns, microscopy images, isotherms, phase diagrams, and property plots.The passage cites demonstrations using MOF sorption isotherms, powder X-ray diffraction patterns, and thermogravimetric analysis curves.Source: §5 · p. 11

5.1 Prompt engineering

Prompt engineering makes LLM extraction adaptable through task instructions, domain context, examples, and output constraints. Reviewed applications use these approaches to build materials databases and interactive systems, while studies also identify trade-offs in accuracy, cost, and workflow design.

5.1 Prompt engineering

  • Prompting strategies include zero- and few-shot in-context learning, chain-of-thought for resolving information across sources, and RAG that retrieves relevant material into the prompt.Domain-specific context and curated examples can reduce hallucination and outperform generic prompting; prompt development is iteratively evaluated against expert-annotated examples.Source: §5.1 · p. 12, Figure 3 · p. 12, §5.1.1 · p. 13

5.1.2 Databases extracted using prompt engineering

  • An LLM prompting workflow extracted more than 100,000 multicomponent-alloy records at ∼0.9 F1, while ChatExtract achieved 0.9 F1 on a manually curated benchmark.ChatExtract was applied to 4,029 papers and extracted 10,269 raw high-entropy-alloy yield-strength values.Source: §5.1.2 · p. 13
  • F1 scores of 0.82–0.88 for LitXBench, versus 0.58 for LeMat-Synth, show the reported advantage of a synthesis-centric schema over a composition-centric pipeline.LitXBench links processing steps to materials and returns executable Python objects for validation and correction.Source: §5.1.2 · p. 14
  • Benchmark studies found that model size alone did not reliably predict extraction performance, and routing fields to their best-performing model raised overall accuracy to 92.5%.A separate polymer-nanocomposite table study found strict matching of property value, unit, and condition yielded only 0.42 F1.Source: §5.1.2 · p. 14
  • Recursive query clarification improved layered-cathode extraction over naive RAG from 0.922 to 0.937 F1 and from 85.6% to 88.6% accuracy.MOF-ChemUnity combined RAG with knowledge-graph-grounded extraction and reported 94% entity-matching accuracy, 89% property precision, and 97.5% synthesis-procedure accuracy.Source: §5.1.2 · p. 14
  • Prompt-engineered databases have supported conversational tools and downstream discovery, including a MOF system reporting 96.9%, 95.7%, and 87.5% accuracies for search, property prediction, and structure generation.NEMAD data trained magnetic-material classifiers and temperature regressors, while extracted perovskite records supported screening 20,000 candidates for synthesizability.Source: §5.1.2 · pp. 14–15

5.1.3 Thumb-rules for better prompt engineering

  • Practical guidance recommends explicit roles and task definitions, rigorous output schemas, smaller verified subtasks, and diverse examples covering absent or ambiguous targets.Schemas should specify field types, units, permissible ranges, and how missing values are represented.Source: §5.1.3 · p. 15
  • Verification-heavy prompting increases API calls, monetary cost, and latency; chain-of-thought uses more tokens, RAG needs added infrastructure, and few-shot examples trade performance against API cost.Iterative prompt refinement also requires domain-expert time.Source: §5.1.3 · pp. 15–16

5.2 Fine-tuning of generative LLMs

Fine-tuning transfers materials-specific terminology and extraction behavior into model parameters, addressing prompting’s domain-knowledge and cost constraints. Reported applications range from joint extraction to large database construction, with advantages strongest for specialized tasks.

5.2 Fine-tuning of generative LLMs

  • Prompting can struggle with specialized notation and become costly at scale, whereas fine-tuning can improve zero-shot schema adherence while reducing prompt length and inference costs.Many-shot prompting may require hundreds or thousands of examples, consume context windows, and bury relevant information.Source: §5.2 · p. 16
  • Fine-tuning transfers materials terminology and task knowledge into model parameters, while expert-annotated data often remain the primary bottleneck.Domain-adaptive pretraining uses unlabeled in-domain text; supervised fine-tuning uses labeled prompt-completion pairs to teach extraction and output schemas, while LoRA and QLoRA reduce adaptation costs.Source: §5.2 · p. 16
  • ∼0.82 was the best dopant F1 score after GPT-3 and Llama-2 were fine-tuned on ∼500 annotated pairs for joint NER and relation extraction across three schemas.The study released its annotations and extracted JSON records; other applications include solar-cell datasets, substitution mining, and property prediction.Source: §5.2 · p. 16
  • 78,203 records covering 19,058 unique compositions formed a superconductivity database used to fine-tune models for classification, Tc prediction, and conditional generation.LLaMat, pretrained on ∼30B tokens of materials text and crystallographic data, reportedly outperformed substantially larger general-purpose commercial models on structured extraction tasks.Source: §5.2 · p. 16
  • Prompted commercial LLMs can be simpler and more accessible, while fine-tuning is more advantageous for highly specialized tasks such as crystal structure generation.The review also cites fine-tuned models for high-entropy-alloy phase classification and molecular solubility.Source: §5.2 · p. 16

5.3 Agentic systems

Materials information is distributed across sections and modalities, making complete records difficult to extract with a single model call or fixed sequence. Agentic systems coordinate adaptive, multi-step retrieval and specialized tools, and have been used for linked databases, multimodal extraction, and closed-loop discovery.

5.3 Agentic systems

  • Materials records require linking information across sections and modalities, while a single article-wide pass or fixed-size chunks can conflate decisions or destroy cross-section dependencies.Composition, synthesis conditions, properties, sample names, and abbreviations may be distributed across article sections, figures, tables, and supplementary information.Source: §5.3 · pp. 16–17

5.3.1 Architecture and design choices of agentic systems

  • Agentic systems coordinate agents, tools, models, and external memory in adaptive multi-step processes rather than relying on one inference or a fixed sequence.Design choices include control algorithm, autonomy, task specialization, available tools, and memory; ReAct interleaves reasoning, tool calls, and observations, while higher autonomy helps when information locations or operation sequences are unknown in advance.Source: §5.3 · p. 17, §5.3.1 · pp. 17–18

5.3.2 Databases produced by agentic pipelines

  • 18,869 temperature-resolved, unit-normalized records were extracted from roughly 104 thermoelectric articles; ALLMAT produced 5,393 records with F1 scores from 0.89 for grain size to 0.98 for yield strength.These pipelines link multiple entities and processing histories; ComProScanner links mechanical properties and phase information to processing histories, while external memory can coordinate partial records and provenance across steps.Source: §5.3.1 · p. 18, §5.3.2 · p. 19
  • ChemEAGLE’s agentic planner improved end-to-end F1 from 0.39 to 0.76, while nanoMINER’s vision component recovered substrate concentrations reported only in plots.Other multimodal pipelines combine text, tables, figures, and molecular or reaction graphics to link document-level evidence.Source: §5.3.2 · p. 19

5.3.3 From extraction to autonomous discovery: Co-scientists and closed-loop agents.

  • 36 of 57 targeted novel inorganic compounds were realized by A-Lab, which iterated robotic synthesis and characterization using model-proposed recipes.Agent-driven systems also span physical experiments, human-validated co-scientist proposals, and in-silico design; LLMatDesign reached a target band gap in an average of 10.8 modifications versus 27.4 for random sampling.Source: §5.3.3 · p. 20

5.3.4 Thumb-rules for better agentic system

  • Extraction agents can be divided among retrieval, modality-specific extraction, normalization, and verification, with extraction separated from schema formatting to improve recall.Use deterministic tools for operations such as arithmetic, canonicalization, and schema validation; match model strength to task complexity, start with a small agent system, evaluate changes with benchmarks, and log intermediate actions for traceability.Source: §5.3.4 · p. 21

5.4 Knowledge graphs: structuring extracted knowledge for reasoning

Knowledge graphs represent extracted materials knowledge as typed entities and relations, enabling queries and reasoning across linked records. Graph embeddings can rank missing relationships, while graph applications span retrieval, question answering, link prediction, and hypothesis generation.

5.4 Knowledge graphs: structuring extracted knowledge for reasoning

  • A knowledge graph represents materials entities as nodes and typed relationships as directed edges, organizing literature as subject-relation-object triples rather than flat rows.Incoming triples instantiate or connect typed nodes, and DOI links preserve provenance; graph queries can retrieve connected properties or applications and trace claims to source records.Source: §5.4 · pp. 21–22

5.4.1 Advantages of knowledge graphs

  • Knowledge graphs support multi-hop queries and can supply LLMs with connected facts and provenance, while graph traversal can derive quantities and embeddings can rank plausible missing links.Ontology construction remains largely human-in-the-loop because automated proposals can introduce bias or diverge from community standards; pairwise triples can also lose composition-processing-property context, motivating hypergraphs.Source: §5.4 · p. 22, §5.4.1 · p. 23

5.4.2 Knowledge graph embeddings

  • Knowledge graph embeddings learn entity and relation vectors from observed and corrupted triples, then score candidate entities for queries with a missing head or tail.Typical models are transductive and cannot directly embed unseen entities; TransE’s translation representation has difficulty with one-to-many or mutual relations, motivating more expressive alternatives.Source: §5.4.2 · pp. 23–24

5.4.3 Applications enabled by knowledge graphs

  • 91.67% QA accuracy was reported for an LLM answering questions from a reticular-materials graph, with supporting triples provided as provenance.Graph applications also include large-scale knowledge organization, while MatKG-2 reported TransE mean reciprocal rank of 0.77 and FMKG found 48.5% of predictions from 2014-era data were reported within nine years.Source: §5.4.3 · p. 24
  • Graph paths can scaffold LLM-generated research hypotheses, and a feedback-loop approach co-evolves the graph and inference process as new concepts and relations are added.The described iterative process produced graphs with emergent scale-free topology and hub formation.Source: §5.4.3 · p. 24

6 Challenges and Opportunities

Materials data extraction still faces challenges in linking terminology across papers, capturing complete sample-level records, and reliably processing multimodal content. The section also identifies gaps in benchmarks and literature access, publication bias, and the need to use extracted databases to generate and validate discoveries.

6 Challenges and Opportunities

  • Inconsistent materials terminology can prevent records for the same entity from linking across papers, limiting cross-paper trend analysis.The review points to canonical identifiers and more standardized reporting as promising directions, while noting that a general, robust solution is still needed.Source: §6 · p. 25
  • Most extracted databases capture only two or three parts of the structure-processing-property-performance tetrahedron, limiting their value for downstream discovery.A complete record should link all four aspects for an individual sample, including its synthesis, processing, resulting structure, property, and performance.Source: §6 · p. 25
  • Multimodal extraction remains more challenging and costly than text extraction, with heterogeneous tables and compound figures complicating reliable plotted-data extraction.The review proposes staged identification, numerical extraction, semantic interpretation, and verification rather than relying on a general-purpose vision-language model for the entire process.Source: §6 · p. 25
  • Small, task-specific datasets hinder performance comparisons, while inaccessible literature and difficult-to-process PDFs constrain automated mining.The review calls for broad expert-annotated benchmarks and wider mining licenses and machine-readable full text.Source: §6 · p. 25
  • Publication bias leaves failed syntheses, negative or null results, and poor-performing materials underrepresented, although supplementary files may contain some of this information.The review identifies dedicated venues and data-sharing policies as potential ways to address the gap.Source: §6 · p. 25
  • The larger goal is to use extracted databases to generate actionable insights or novel materials and validate them experimentally, rather than ending at database construction.The review describes LLM agents grounded in materials knowledge graphs and coupling databases to autonomous laboratories as directions toward hypothesis validation and closed-loop discovery.Source: §6 · p. 25

Related papers

Loading 2610.29895v1…