PaperlayerView exact revision on arXiv

Source-linked AI summary

MatSciBERT: A Materials Domain Language Model for Text Mining and Information Extraction

Tanishq Gupta, Mohd Zaki, N. M. Anoop Krishnan, Mausam

arXiv:2109.15290v1cs.CLcond-mat.mtrl-sci

TL;DR

Materials literature contains valuable information, but general language models may miss domain-specific notation and jargon. The paper develops MatSciBERT by further pretraining SciBERT on materials literature and evaluates it across materials tasks, where it outperforms SciBERT on all tested datasets.

  • Problem

    Materials texts contain domain-specific notation and jargon, while existing language models lack materials-specific training.

  • Method

    MatSciBERT is trained by further pretraining SciBERT on a corpus spanning representative materials science literature.

  • Results

    MatSciBERT outperforms SciBERT across document classification, named entity recognition, and relation classification datasets.

  • Takeaways & Limitations

    MatSciBERT supports automated information extraction from materials science text corpora and related materials applications.

  • Takeaways & Limitations

    Few annotated materials datasets are available, making dataset development important alongside materials-specific language-model development.

Abstract

from arXiv · show

An overwhelmingly large amount of knowledge in the materials domain is generated and stored as text published in peer-reviewed scientific literature. Recent developments in natural language processing, such as bidirectional encoder representations from transformers (BERT) models, provide promising tools to extract information from these texts. However, direct application of these models in the materials domain may yield suboptimal results as the models themselves may not be trained on notations and jargon that are specific to the domain. Here, we present a materials-aware language model, namely, MatSciBERT, which is trained on a large corpus of scientific literature published in the materials domain. We further evaluate the performance of MatSciBERT on three downstream tasks, namely, abstract classification, named entity recognition, and relation extraction, on different materials datasets. We show that MatSciBERT outperforms SciBERT, a language model trained on science corpus, on all the tasks. Further, we discuss some of the applications of MatSciBERT in the materials domain for extracting information, which can, in turn, contribute to materials discovery or optimization. Finally, to make the work accessible to the larger materials community, we make the pretrained and finetuned weights and the models of MatSciBERT freely accessible.

1 Introduction

Materials knowledge is abundant in scientific text but difficult to extract because literature is extensive and largely unstructured. The paper addresses the lack of a materials-aware language model by introducing MatSciBERT for downstream information-extraction tasks.

  • Motivation: Materials discovery is slow, while relevant knowledge is distributed across a literature corpus too large for manual extraction.Scientific information is stored in publications, reports, handbooks, and websites, but reviewing all available literature is humanly impractical.
  • Motivation: Experimental procedures and findings are often reported in paragraphs, captions, graphs, microscopy images, and diffraction results.
  • Prior work: NLP pipelines have already extracted battery materials, magnetic transition temperatures, and inorganic synthesis routes from scientific text.
  • Prior work: Existing tools support chemical parsing, synthetic-route generation, robotic thin-film production, and identification of materials-specific named entities.
  • Research gap: The materials domain lacks a materials-aware language model that can adapt effectively to downstream tasks.
  • Contribution: MatSciBERT targets NER on SOFC and Matscholar, glass classification, and relation classification on the MSPT corpus.

2 Methodology

The methodology creates a materials science corpus, trains MatSciBERT, and evaluates it on downstream tasks.

  • Methodology: The workflow begins by creating a materials science corpus.
  • Methodology: MatSciBERT is trained using the constructed materials science corpus.
  • Methodology: The trained model is evaluated on different downstream tasks.

2.1 Dataset collection and preparation

The Materials Science Corpus covers four representative materials categories and is assembled from large-scale literature retrieval, manual annotation, classifier filtering, and Unicode normalization.

  • Corpus scope: The corpus covers inorganic glasses and ceramics, metallic glasses, cement and concrete, and alloys.
  • Collection: Crossref retrieval produced more than 1M articles, from which papers were downloaded through Elsevier Science Direct’s sanctioned API.
  • Filtering: Researchers manually annotated 500 abstracts and fine-tuned one SciBERT classifier per materials category to identify relevant papers.
  • Normalization: Unicode normalization removes random characters and maps visually or semantically similar characters to standard representations.
  • Normalization: The normalized corpus maps random characters to spaces and applies the same normalization to every downstream dataset before MatSciBERT processing.

2.2 Pre-training of MatSciBERT

MatSciBERT is initialized from SciBERT and further pretrained on materials literature using the SciBERT vocabulary and a modified RoBERTa training recipe.

  • Initialization: MatSciBERT is initialized with SciBERT weights and tokenized using SciBERT’s uncased vocabulary.This approach reduces the computational burden of training from scratch and supports interchangeability with existing SciBERT-based models.
  • Pretraining recipe: Pretraining uses dynamic whole-word masking, removes the NSP loss, uses full-length sequences, and increases batch sizes.
  • Pretraining recipe: MatSciBERT is pretrained for 10 days on two NVIDIA V100 32GB GPUs with a batch size of 256 sequences per GPU and maximum sequence length 512.

2.3 Downstream Tasks

The paper evaluates MatSciBERT on named entity recognition, abstract classification, and relation classification using materials-focused datasets. These tasks cover entity identification, field relevance classification, and directed relationships between non-overlapping entity spans.

  • Tasks: MatSciBERT is evaluated on named entity recognition, text/abstract classification, and relation classification.The downstream evaluation adapts the pretrained language model to task-specific supervised settings.
  • Named Entity Recognition: NER identifies domain-specific entities in tokenized sentences using BIO labels for multi-token entities.Labels distinguish entity beginnings, continuations, and non-entity tokens across dataset-specific entity types.
  • Relation Classification: Relation classification maps a sentence and an ordered pair of non-overlapping entity spans to a directed relationship label.The spans are represented by their start and end indices, and the output belongs to a fixed relation-type set.
  • Text/Abstract Classification: Text/abstract classification determines whether a research-paper abstract is relevant to a specified field.The evaluation uses a glass-science versus non-glass classification dataset.
  • Datasets: The NER evaluation uses Matscholar, SOFC, and fine-grained SOFC-Slot datasets with differing entity inventories and annotations.SOFC-Slot contains 16 primary fine-grained entity types plus Experiment Evoking Word and Thickness for model training.
  • Datasets: Relation classification uses 230 annotated synthesis procedures, while abstract classification uses 1500 manually labelled glass and non-glass abstracts.The synthesis dataset represents procedures as graphs whose nodes are participants and whose edges specify relationships.

2.4 Modelling

The modelling section applies BERT-based architectures tailored to token, abstract, and entity-pair prediction. It compares SciBERT and MatSciBERT using task-specific output layers and standardized evaluation metrics.

  • Named Entity Recognition: NER uses contextual embeddings for each token with LM-Linear, LM-CRF, and LM-BiLSTM-CRF architectures.LM denotes either SciBERT or MatSciBERT; the CRF-based models learn entity-label transitions.
  • Relation Classification: Relation classification marks two entity spans, concatenates their boundary embeddings, and predicts the relation through a softmax linear layer.The linear layer and language model are fine-tuned with standard cross-entropy loss.
  • Text/Abstract Classification: Abstract classification uses the CLS embedding as a representation of the entire text before a simple classifier predicts its class.The implementation uses the BERT Sentence Classifier from the transformers library.
  • Optimization: Training uses a linear-decay learning-rate schedule with 0.1 warmup, AdamW, batch size 16, and separate learning rates for BERT and non-BERT layers.The non-BERT peak learning rate is 3e-4, while the BERT rate is selected from 2e-5, 3e-5, and 5e-5 using validation performance averaged across 3 seeds.
  • Evaluation: NER is evaluated by entity-level exact match, while Micro-F1 and Macro-F1 measure NER and relation classification and binary F1 measures abstract classification.The CoNLL evaluation script is used for NER after verifying its correctness.

3 Results and discussion

The authors build a materials corpus, initialize MatSciBERT from SciBERT, and pre-train it before evaluating materials-domain tasks. MatSciBERT improves over SciBERT across reported NER comparisons and reaches stronger Matscholar results.

  • 3.1 Dataset: The materials corpus contains approximately 285M words spanning inorganic glasses and ceramics, bulk metallic glasses, alloys, and cement and concrete.The corpus distribution is approximately 40% inorganic glasses and ceramics and 20% each for the other three families.
  • 3.2 Pre-training of MatSciBERT: MatSciBERT is initialized with SciBERT weights and vocabulary because the materials corpus vocabulary overlaps more with SciBERT than with BERT.The reported uncased vocabulary overlaps are 53.64% with SciBERT and 38.90% with BERT.
  • 3.2 Pre-training of MatSciBERT: The final pretrained language model achieves a validation perplexity of 3.112 before downstream evaluation against SciBERT.Figure 2 tracks validation perplexity over pretraining duration, while the downstream comparison tests learning of materials-specific information.
  • 3.2 Pre-training of MatSciBERT: Longer pre-training increases SOFC-Slot NER performance, as shown by the upward Macro-F1 trend over elapsed training days.The experiment periodically fine-tunes the model on SOFC-Slot to assess the effect of pretraining duration.
  • 3.3 Downstream tasks: ~4.2 Macro F1 and ~2.1 Micro F1 improvements are obtained on the SOFC-Slot test set with LM-BiLSTM-CRF relative to SciBERT.MatSciBERT-BiLSTM-CRF also exceeds current best Macro-F1 results by ~3.3 on SOFC-Slot and ~0.6 on SOFC.
  • 3.3 Downstream tasks: 89.33% validation Micro-F1 and 87.54% test Micro-F1 are achieved by MatSciBERT LM-CRF on Matscholar, exceeding the cited state-of-the-art results.The cited prior validation and test values are 87.09% and 87.04%, respectively.
  • 3.3 Downstream tasks: Example predictions show MatSciBERT correctly identifies ND, TD, and YSZ labels that SciBERT misclassifies, although both models misclassify “dislocation cell-like structure.”The examples are explicitly described as arbitrary and not necessarily representative.

Ground truth and model predictions

The paper compares MatSciBERT with SciBERT across materials-focused classification, NER, and relation-classification tasks, then illustrates applications in document and image information extraction. MatSciBERT consistently performs better, while the paper also identifies annotated-data scarcity as an important constraint.

  • Model evaluation: MatSciBERT outperforms SciBERT by ~2.5 F1-score on both validation and test sets for glass-versus-non-glass abstract classification.The task classifies manuscript abstracts into glass and non-glass topics.
  • Model evaluation: MatSciBERT performs better than SciBERT consistently on relation classification, although the improvement margin is lower.The comparison uses the Materials Synthesis Procedures dataset; the reported results are not directly comparable with an earlier study because the task assumptions and relation classes differ.
  • Applications: MatSciBERT uses contextual embeddings to distinguish materials terms whose meanings vary with surrounding text, supporting topic classification.The paper contrasts this with TF-IDF and other approaches that directly vectorize words without context sensitivity.
  • Applications: NER applied to approximately 1,10,000 inorganic-glass image captions extracted entities across sample descriptors, applications, materials, properties, characterization methods, and synthesis methods.Reported counts include 145324 MAT entities, 87318 DSC entities, 76898 PRO entities, 73241 CMT entities, 33426 SMT entities, and 10633 APL entities.
  • Applications: MatSciBERT extracted multiple entities from selected image captions, whereas prior manual annotation assigned only one word per caption.The comparison illustrates the amount of information captured by the NER model in the selected examples.
  • Applications: Relation classification can provide information about sequential materials-science processes, including synthesis, testing, and measurement protocols.The paper also identifies question answering as another potential application when MatSciBERT is trained on a suitable dataset.

4 Conclusion

The paper develops MatSciBERT, a materials-aware language model trained on materials-science journal corpora, and evaluates it on several downstream tasks. It reports superior performance to SciBERT across the tested datasets and discusses applications for information extraction from materials text.

  • 4 Conclusion: MatSciBERT is a materials-aware language model trained on materials-science journal corpora.It starts from SciBERT weights and adds materials-science information.
  • 4 Conclusion: MatSciBERT exhibits superior performance to SciBERT on the tested document-classification, NER, and relation-classification datasets.These evaluations cover several downstream materials-science tasks.
  • 4 Conclusion: The paper discusses applications through which MatSciBERT can support information extraction from materials-science text corpora.The stated application scope is connected to accelerating information extraction in the materials domain.
Loading 2109.15290v1…