PaperlayerView exact revision on arXiv

Source-linked AI summary

Structured information extraction from complex scientific text with fine-tuned large language models

Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S. Rosen, Gerbrand Ceder, Kristin Persson, Anubhav Jain

arXiv:2212.05238v1cs.CLcond-mat.mtrl-sci

TL;DR

Complex scientific information is difficult to extract and link from unstructured text. The paper fine-tunes GPT-3 for sequence-to-sequence entity and relation extraction, reporting strong performance across materials tasks and high sequence-level similarity and parsability.

  • Problem

    Complex hierarchical scientific relationships are difficult to extract because conventional relation extraction requires enumerating many relation types and sufficient examples for each.

    Source: Prior work · p. 3

  • Method

    The authors fine-tune GPT-3 on annotated examples to generate structured summaries from text, including records in user-defined schemas.

    Source: §I · p. 1, General seq2seq NERRE · p. 4

  • Results

    All models have average Jaro-Winkler similarities ≥ 91.7% and parsabilities ≥98.7%, while simpler doping models exactly match 58.4−64.9% of test sequences.

    Source: Sequence-level results · pp. 9–10

  • Takeaways & Limitations

    The method performs strongly across three materials-engineering tasks and is presented as usable by researchers with little NLP experience.

    Source: §V · p. 15

    Valid output formatting is not rigorously enforced, and outputs may be unparsable, particularly with inference limits below 512 tokens for JSON.

    Source: Limitations · p. 14

Abstract

from arXiv · show

Intelligently extracting and linking complex scientific information from unstructured text is a challenging endeavor particularly for those inexperienced with natural language processing. Here, we present a simple sequence-to-sequence approach to joint named entity recognition and relation extraction for complex hierarchical information in scientific text. The approach leverages a pre-trained large language model (LLM), GPT-3, that is fine-tuned on approximately 500 pairs of prompts (inputs) and completions (outputs). Information is extracted either from single sentences or across sentences in abstracts/passages, and the output can be returned as simple English sentences or a more structured format, such as a list of JSON objects. We demonstrate that LLMs trained in this way are capable of accurately extracting useful records of complex scientific knowledge for three representative tasks in materials chemistry: linking dopants with their host materials, cataloging metal-organic frameworks, and general chemistry/phase/morphology/application information extraction. This approach represents a simple, accessible, and highly-flexible route to obtaining large databases of structured knowledge extracted from unstructured text. An online demo is available at http://www.matscholar.com/info-extraction.

I. INTRODUCTION

Materials knowledge is scattered across papers, while conventional extraction often identifies entities without capturing the complex links among them. The paper proposes a flexible GPT-3 sequence-to-sequence approach for joint entity and relation extraction, building on prior work while addressing hierarchical scientific information.

I. INTRODUCTION

  • Materials knowledge is dispersed across papers, and property-prediction models are limited by the amount of training data in tabulated databases.The text notes this limitation is particularly apparent for experimental properties and parameters.Source: §I · p. 1
  • GPT-3 is fine-tuned on approximately 500 document-completion examples to produce structured records for document-level joint entity and relation extraction.Outputs can be English sentences or structured formats such as JSON.Source: §I · p. 1

Prior work

  • Materials information extraction has often targeted entities, but downstream machine learning and knowledge graphs also require relations linking those entities.Prior materials studies extracted entities such as materials or experimental parameters, while relation extraction identifies links between entities.Source: Prior work · p. 2
  • Materials relations can be hierarchical and depend on combinations of composition, geometry, morphology, processing, and environmental factors.Conventional relation extraction is not designed to preserve these complex relationships across arbitrary numbers of entities, and enumerating n-tuple variants requires many examples.Source: Prior work · pp. 2–3

Sequence-to-sequence (seq2seq) relation extraction

  • The proposed seq2seq approach handles complex inter-relations without enumerating every n-tuple relation or performing preliminary named entity recognition.It fine-tunes GPT-3 to turn passages into summaries in a user-defined output structure, using approximately 100–500 annotated passages.Source: Sequence-to-sequence (seq2seq)… · p. 3
  • The approach is presented as accessible to researchers with little NLP experience and does not appear to require fine-tuning on a large domain-specific corpus.The authors report strong performance on sentence-level and document-level materials information extraction.Source: Sequence-to-sequence (seq2seq)… · p. 3

II. METHODS

The methods describe fine-tuning GPT-3 to extract entities and relationships into predefined schemas across three materials-information tasks, using human-in-the-loop annotation to build training data. Evaluation combines sequence reconstruction with information-extraction scoring, assessing entity validity, correct root grouping, and field placement.

General seq2seq NERRE

  • LLM-NERRE fine-tunes GPT-3 on 100–1,000 annotated prompt-completion pairs to produce outputs in the examples’ predefined schema.A partially trained model pre-fills annotations for human correction, reducing average annotation time from 100 to around 40 seconds per abstract; the final model extracts without human correction.Source: General seq2seq NERRE · pp. 3–4

Benchmark tasks and output schema design

  • Doping schemas represent host–dopant relations and may also extract composition results and modifiers, while sentence-level training reflects where most doping information occurs.Doping-JSON encodes relations in JSON and Doping-ENG in quasi-natural-language summaries; DopingExtra-ENG additionally outputs results and modifiers.Source: Solid-state impurity doping schema · pp. 4–6
  • General-JSON creates one JSON entry per named or formulated solid material, with fields for descriptions, structure or phase, and applications.The schema also captures acronyms and defines structure/phase to include crystal systems, structure names, space groups, unit-cell information, and amorphous labels.Source: General materials information schema · p. 6
  • MOF-JSON adapts the general materials schema to catalog MOF names or formulas, guest species, applications, and sample descriptors.Entries are generally ordered by the appearance of material names or formulas in the text.Source: Metal–organic framework (MOF) schema · p. 6

Comparison benchmarks and evaluation

  • The doping comparison benchmarks use seq2rel for sentence-level extraction and MatBERT named-entity recognition followed by same-sentence proximity linking.The seq2rel data match the Doping-JSON and Doping-ENG training data; the MatBERT model was fine-tuned on 455 manually annotated abstracts.Source: Comparison benchmarks and evaluation · pp. 6–7

Datasets

  • Training data came from research-paper abstracts: 162 abstracts for doping, approximately 650 annotated general-materials entries, and 507 expert-annotated MOF abstracts.Doping evaluation used a separate holdout set of 232 sentences from 31 abstracts; general-materials and MOF evaluations used random 10% validation samples averaged over five trials.Source: Datasets · p. 7, Doping dataset · p. 7, General materials dataset · p. 8

GPT-3 fine tuning details

  • For all tasks, GPT-3 davinci was fine-tuned at batch size 1; doping models ran 7 epochs, while General-JSON and MOF-JSON ran 4 epochs.Inference temperature was 0; output limits were 256 tokens for doping and 512 for General-JSON and MOF-JSON. The learning-rate multiplier (0.1) and prompt-loss weight (0.01) were not hyperparameter-tuned.Source: GPT-3 fine tuning details · p. 8

Evaluation criteria

  • Evaluation measures sequence reconstruction and joint entity–relation extraction, using exact word-match triplet F1 and a domain expert’s manual information-extraction score for valid representations.Formula-bearing entities and their relation triplets require exact composition matches; triplet evaluation covers selected relations rather than the full task of linking all interrelated entities.Source: Evaluation criteria · p. 8, Information extraction performance · pp. 8–9
  • An entity had to come from the passage or be a reasonable variation, not a model invention.For example, “silicon” could be normalized to “Si.”Source: Information extraction performance · p. 9
  • An entity had to be a root entity or be grouped with a valid root; for General-JSON, the root was the material’s formula or name, using the formula when both appeared.Source: Information extraction performance · p. 9
  • An entity had to occupy the correct field in the correct root entity’s JSON object.Source: Information extraction performance · p. 9
  • Although manual extraction scores were reported per entity like NER scores, a true positive implicitly required correct relational grouping with the root entity.Source: Information extraction performance · p. 9

III. RESULTS

LLM-NERRE models outperform the two baselines on doping extraction, while complex outputs are less likely to match exactly despite high similarity and parsability. Results also show stronger manually evaluated extraction scores and only marginal, noisy improvement above 64 training samples.

Sequence-level results

  • 58.4–64.9% of test sequences were exact matches for simpler doping models, versus 12.5–25.6% for MOF-JSON and General-JSON.Across models, average Jaro-Winkler similarities were ≥ 91.7% and parsabilities were ≥98.7%.Source: Sequence-level results · pp. 9–10, Table 2 · p. 10
  • With 20+ entities, exact matches were very unlikely, while sequence similarity and parsability remained high for General-JSON and MOF-JSON outputs.Exact-match sequence reconstruction was highest for the simplest entries, which contain the fewest entities.Source: Figure 6 · p. 10, Sequence-level results · p. 10

Relation extraction performance

  • All three LLM-NERRE models exceeded both baselines in NERRE and pure NER performance despite training on 413 sentences versus 455 abstracts.Doping-model F1 exceeded 0.7, with DopingExtra-ENG performing best.Source: Doping task · p. 11
  • General-JSON NERRE F1 ranged from 0.613 for formula→name to 0.341 for formula→description; MOF guest-species links reached 0.606, while name-to-description links were least reliable.The authors suggest fuzzier annotation boundaries for descriptions may contribute to lower fidelity.Source: General and MOF tasks · p. 11, Table 4 · p. 12, Table 5 · p. 12
  • Manual review raised General-task information-extraction F1 to 0.943 for formulae, 0.848 for materials names, 0.832 for applications, 0.829 for structures, and 0.704 for descriptions.These manually evaluated scores were based on a 10% random subset of the test set and account for ambiguity in equivalent wording.Source: General and MOF tasks · p. 11, Table 6 · p. 12
  • Acronyms had the lowest manual extraction scores; they appeared in only 52 abstracts (about 9% of documents) and can be confused with chemical formulae.The authors suggest that additional acronym training data may improve extraction scores.Source: General and MOF tasks · p. 11

Effect of training set size

  • Doping-ENG produced no parsable sequences below 8 training samples, increased sharply at 16, and showed only marginal, noisy improvement above 64.Each data point represented one independent training/prediction experiment; internal model variability may explain score variance at higher sample counts.Source: Effect of training set size · p. 12

IV. DISCUSSION

LLM-NERRE offers an accessible way to extract intricate, hierarchical scientific information and can use intermediate model suggestions to speed annotation. Its outputs can normalize entities, but schema formatting, input length, and API access constrain use.

Accessibility and ease-of-use

  • Domain specialists can define a schema, annotate examples, and fine-tune through public APIs with little NLP expertise, while flexible schemas support intricate hierarchical knowledge extraction.This contrasts with methods that enumerate possible entity relationships and depend more heavily on output-format tuning and examples for each relation type.Source: §IV · pp. 12–13, Accessibility and ease-of-use · p. 13
  • With 300 training samples, the intermediate General-JSON model reduced annotation time per abstract by 57% versus the 1-sample model, while models trained on more than 50 samples required substantially less correction.At 300 samples, annotation time per token was about 60% lower than with one sample and 38% above verification time.Source: Accessibility and ease-of-use · pp. 13–14

Implicit error correction and entity normalization

  • General-JSON training examples can teach implicit correction and normalization, such as resolving “Li Co O2” to “LiCoO2” without post-processing.With sufficient examples, the model can also normalize “PdO functionalized with platinum” to a formula and description format.Source: Implicit error correction and entity… · p. 14

Limitations

  • Valid output formatting is not enforced, and some long or information-dense abstracts exceed GPT-3’s 2048-token limit, truncating completions; outputs are nearly always parsable, at about 95%.Formatting failures are particularly apparent for JSON with inference limits below 512 tokens.Source: Limitations · p. 14
  • LLMs can generate information absent from the text, while normalized entities may have multiple valid forms that complicate downstream disambiguation.The authors prefer normalization despite this variability, noting that exact-phrase extraction can create post-processing complications.Source: Limitations · pp. 14–15
  • GPT-3 is available only through OpenAI’s online API, requiring data to be sent to a private entity; inference costs on large datasets may be prohibitive.The authors expect sufficiently capable open-source sequence-to-sequence models could reproduce or exceed these results.Source: Limitations · p. 15

V. CONCLUSION

The paper presents prompt- and completion-engineering with large sequence-to-sequence models for extracting hierarchical relations from scientific text. It reports strong performance on three materials-engineering tasks and describes the approach as usable by scientists without NLP training.

V. CONCLUSION

  • The method extracts complex, hierarchical, domain-specific relations from scientific text using large sequence-to-sequence language models.The authors report strong performance on solid-state impurity doping, metal-organic frameworks, and general materials relations.Source: §V · p. 15

VI. AUTHOR CONTRIBUTIONS

The authors divide responsibilities across method development, dataset and schema creation, model training, experiments, scoring, and supervision.

VI. AUTHOR CONTRIBUTIONS

  • Contributors developed the extraction methods and schemas, collected and annotated datasets, trained models, ran experiments and scoring, and supervised and wrote the work.Responsibilities included doping, general materials, and MOF tasks, plus learning-curve and annotation-time experiments.Source: §VI · p. 15

VIII. SUPPLEMENTARY INFORMATION

Supplementary discussion describes how annotation ambiguity constrains extraction, particularly for long-range doping links and inconsistently defined general-task entities. The general schema is not optimized for composites, although observed annotation inconsistencies did not significantly impair the model’s composite handling.

Ambiguity of annotations

  • LLM-NERRE performance depends on consistently and comprehensively defining schemas and entities, which can be ambiguous even for trained annotators.The authors identify reducing schema ambiguity as a potential improvement.Source: Ambiguity of annotations · p. 18
  • Dopant–host links can be unclear for long-range relationships, motivating sentence-level rather than abstract-level annotation for the doping task.Most sentence-level doping relationships were clear to annotators.Source: Ambiguity of annotations · p. 18
  • General-JSON class definitions drifted during annotation, and the schema was not optimized for composites; inconsistent composite labels lowered string-match F1 but did not significantly impair composite handling.Composite components were sometimes annotated separately and sometimes as a single material.Source: Ambiguity of annotations · p. 18

Doping-ENG schema

This section defines the English doping-output schema and sequence metrics, reports entity-recognition differences, and illustrates the hierarchical graph output and evaluation-set scope.

Doping-ENG schema

  • The English schema represents host–dopant links, unlinked hosts or dopants, possible doped-result formulas, and one or more doping modifiers.Outputs that do not match a listed pattern are considered unparsable.Source: Doping-ENG schema · p. 18
  • A separate schema pattern marks sentences with no doping-related information.Source: Doping-ENG schema · p. 18

Sequence reconstruction metrics

  • Exact match requires character-for-character sequence agreement; Jaro-Winkler similarity compares predicted and true strings, while parsability measures whether outputs can be transformed into relational objects.Exact match is stringent because whitespace or entry-order differences count as errors.Source: Sequence reconstruction metrics · pp. 18–19

Performance on named entity recognition

  • Doping-ENG had host F1 1% higher than DopingExtra-ENG, while DopingExtra-ENG had dopant F1 8% higher; Doping-JSON F1 was 7–9% lower than Doping-ENG.The authors speculate that the difference between the English models may reflect model variability and extra specification of non-dopant chemical species in DopingExtra-ENG training.Source: Performance on named entity recognition · p. 19

GPT-3 inference parameters

  • Doping models used 7 epochs; other models generally used batch size 1 for 4 epochs, while Figure 7 intermediates used 2, 4, or 7 epochs depending on training-set size.Source: GPT-3 inference parameters · p. 20

seq2rel parameters

  • The seq2rel comparison used 267 linked doping sentences, four train-validation splits, and the model with the highest validation micro-F1.Its configuration included max_length=512, max_steps=96, 30 epochs, batch_size=1, and beam_size=4.Source: seq2rel parameters · p. 20

Doping task graph example

  • The example graph encodes hierarchical relationships, including dopant and application links, as a step toward the paper’s intended final product.The graph can distinguish ZnO nanoparticles from bulk wurtzite ZnO and represent the samarium dopant’s oxidation state and amount.Source: Doping task graph example · p. 20, Figure S1 · p. 20

Class support for materials extraction tasks

Related papers

Loading 2212.05238v1…