Source-linked AI summary
Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering
Maciej P. Polak, Dane Morgan
TL;DR
Automated materials-data extraction has traditionally required substantial setup, expertise, and coding. This paper presents ChatExtract, a zero-shot conversational-LLM workflow using engineered prompts and follow-up verification. It reports high-quality extraction for bulk modulus and critical cooling-rate data, alongside databases for metallic glasses and high-entropy alloys.
Problem
Earlier automated research-paper extraction methods require substantial setup, coding, expertise, or task-specific training data.
Method
ChatExtract uses zero-shot conversational LLMs with engineered prompts to identify relevant sentences, extract Material, Value, Unit triplets, and verify them through follow-up questions.
Results
ChatExtract achieved 90.8% precision and 87.7% recall for bulk modulus, and 91.6% precision and 83.6% recall for critical cooling rates.
Takeaways & Limitations
The method produced databases of critical cooling rates for metallic glasses and yield strengths for high entropy alloys.
Abstract
from arXiv · showhide
There has been a growing effort to replace manual extraction of data from research papers with automated data extraction based on natural language processing, language models, and recently, large language models (LLMs). Although these methods enable efficient extraction of data from large sets of research papers, they require a significant amount of up-front effort, expertise, and coding. In this work we propose the ChatExtract method that can fully automate very accurate data extraction with minimal initial effort and background, using an advanced conversational LLM. ChatExtract consists of a set of engineered prompts applied to a conversational LLM that both identify sentences with data, extract that data, and assure the data's correctness through a series of follow-up questions. These follow-up questions largely overcome known issues with LLMs providing factually inaccurate responses. ChatExtract can be applied with any conversational LLMs and yields very high quality data extraction. In tests on materials data we find precision and recall both close to 90% from the best conversational LLMs, like ChatGPT-4. We demonstrate that the exceptional performance is enabled by the information retention in a conversational model combined with purposeful redundancy and introducing uncertainty through follow-up prompts. These results suggest that approaches similar to ChatExtract, due to their simplicity, transferability, and accuracy are likely to become powerful tools for data extraction in the near future. Finally, databases for critical cooling rates of metallic glasses and yield strengths of high entropy alloys are developed using ChatExtract.
I. INTRODUCTION
ChatExtract addresses the effort and expertise required by earlier automated extraction methods through zero-shot conversational LLMs and engineered prompts. Its workflow combines data identification, extraction, and follow-up verification to improve accuracy and transferability.
- Motivation: Earlier automated extraction methods often require parsing rules, model fine-tuning, retraining, or substantial training-data preparation.These requirements can demand significant time, resources, coding, and domain expertise.
- Motivation: Conversational LLMs offer opportunities for materials-data extraction with almost no initial effort because they are broadly capable and pretrained for general tasks.
- ChatExtract: ChatExtract uses zero-shot conversational LLMs and engineered prompts to extract materials properties as Material, Value, Unit triplets.
- ChatExtract: Follow-up questions check extracted details, introduce uncertainty, and add redundancy to reduce extraction errors and hallucinations.
- Scope and transferability: ChatExtract is designed to be flexible and efficient across information-extraction tasks, although prompt engineering may need modification for different data types.
A. Description of the Data Extraction Workflow
The ChatExtract workflow first filters sentences for relevance, then extracts structured materials data with prompts tailored to single- and multi-valued sentences. Conversational context, explicit negative responses, redundancy, and uncertainty-inducing verification support automated quality control.
- Stage (A): relevance classification: Stage (A) applies a relevancy prompt to all sentences to remove those that do not contain the target materials data.The workflow then expands each relevant sentence into a passage containing the title, preceding sentence, and target sentence.
- Stage (B): data extraction: The workflow separates single-valued from multi-valued sentences because multi-valued text requires analyzing relations among materials, values, and units.Single-valued text generally needs simpler extraction, whereas multi-valued text is more prone to errors and requires further verification.
- Stage (B): data extraction: Prompts explicitly allow missing data and negative answers, discouraging hallucinated values and discarding sentences when required information is absent.
- Stage (B): verification: The prompts keep questions in one conversation, repeat the full data in each prompt, and enforce Yes/No answers for information retention and automation.
- Stage (A): relevance classification: The expanded passage helps include the material name, which may appear in the preceding sentence or paper title rather than the target sentence.
- Stage (B): verification: Multi-valued sentences are converted into structured tables, after which each field is checked with redundant follow-up questions that introduce doubt about possible inaccuracies.
B. Performance Evaluation and Model Comparison
ChatExtract achieved strong zero-shot bulk-modulus extraction performance, while follow-up prompts and conversational history materially supported precision and recall. Alternative models also extracted data, but GPT-based prompts and models remained unevenly transferable.
- Bulk-modulus performance: 90.8% precision and 87.7% recall were achieved by ChatGPT-4 in the bulk-modulus test without fine-tuning.Single-valued sentences reached 100% recall, compared with 82.7% for multi-valued sentences.
- Mechanisms: Redundant prompts introduce uncertainty about extracted data, while conversational retention lets follow-up questions use previous responses.The authors identify these as two core features supporting factual correctness and precision.
- Ablation findings: Removing follow-up questions reduced overall precision from 90.8% to 42.7% for ChatGPT-4 and from 70.1% to 26.5% for ChatGPT-3.5.The additional extracted triplets mainly introduced false positives among passages without data.
- Ablation findings: 10.7 percentage points of overall recall were lost when ChatGPT-3.5 processed each prompt in a new conversation without retained history.Precision showed little or no reduction in this no-chat test.
- Model comparison: LLaMA2-chat achieved 61.5% precision and 62.9% recall overall, slightly below ChatGPT-3.5.The prompts were optimized using GPT models, so further model-specific optimization could affect this comparison.
C. Application to Tables and Figures
ChatExtract was extended from sentence extraction to tables and figures. Table extraction produced high classification and datapoint-level performance, whereas figure evaluation was complicated by data embedded indirectly in plots or absent despite relevant captions.
- Tables: Tables and captions were classified for relevance, then relevant table text and captions were supplied to the LLM for structured extraction.The workflow targets only the specified property and returns the relevant data in table form.
- Figures: Figure processing is treated differently because figure analysis is not a language-processing task and remains challenging for artificial intelligence.LLMs can nevertheless help identify relevant figures for subsequent human analysis.
- Figures: Figures showing fitted equations of state were often classified as relevant even when the bulk-modulus value was not directly present.Such cases reduced precision, while figures considered relevant despite indirect presentation also negatively affected recall.
- Database applications: ChatExtract was used to build databases of critical cooling rates for bulk metallic glasses and yield strengths for high-entropy alloys.The directly extracted raw database can retain ranges, limits, approximate values, material families, and outliers.
D. Results of Real-life Data Extraction
ChatExtract was applied to build critical-cooling-rate and high-entropy-alloy yield-strength databases, with results supporting transfer across properties while revealing limits in standardization and figure extraction.
- Two materials-property databases were developed: critical cooling rates for metallic glasses and yield strengths for high entropy alloys.
- Standardization combined manual work, LLM prompting, regular expressions, and pymatgen, but required substantial additional prompt engineering and coding.
- 348 raw table datapoints added 277 new standardized points and 97 new unique compositions to the critical-cooling-rate database.
- 557 datapoints remained in the final standardized critical-cooling-rate database, including 222 unique material compositions after duplicate handling.
- The HEA yield-strength database was significantly larger than earlier HEA databases, including those with 169 and 419 unique compositions.
E. Conclusions
The paper concludes that conversational LLMs with engineered prompts and follow-up questions can extract materials data accurately without fine-tuning or extensive coding. It reports strong benchmark performance and databases for two materials properties.
- ChatExtract extracts materials data with over 90% precision and 87.7% recall on bulk modulus, and 91.6% precision and 83.6% recall for critical cooling rates.
- Purposeful redundancy, uncertainty-inducing follow-up questions, and conversational information retention account for ChatExtract's success.
- The method requires no additional fine-tuning, extensive code development, or deep property-specific knowledge.
- ChatExtract was used to develop critical-cooling-rate and high-entropy-alloy yield-strength databases.
III. METHODS
Performance was assessed using precision and recall under passage-level matching rules for material–value–unit triplets.
- Precision and recall were the main statistical quantities used to assess ChatExtract performance.
- True positives and false negatives were defined using passages containing a target sentence, its preceding sentence, and the title.
- Equivalent triplets required identical units and values, while material names had to identify the same materials system.
- The extracted databases and assessment data were made available through Figshare.