ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis

Zhiling Zheng; Oufan Zhang; Christian Borgs; Jennifer T. Chayes; Omar M. Yaghi

Source-linked AI summary

ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis

Zhiling Zheng, Oufan Zhang, Christian Borgs, Jennifer T. Chayes, Omar M. Yaghi

arXiv:2306.11296v2Jun 20, 2023cs.IRcond-mat.mtrl-scics.CLphysics.chem-ph

TL;DR

MOF synthesis information is difficult to mine reliably from diverse scientific literature formats. The paper introduces a prompt-engineered ChatGPT workflow that extracts structured conditions, achieving high extraction performance across three processes.

Problem
Diverse reporting formats make locating and standardizing MOF synthesis conditions and organic-linker representations difficult.
Method
The study develops three prompt-engineered ChatGPT processes for locating, classifying, summarizing, and unifying MOF synthesis information, plus a random-forest crystallization predictor.
Results
All three processes achieved precision >95%, recall >90%, and F1 >92%, with 0 cases of hallucination detected by human evaluators.
Takeaways & Limitations
The ChatGPT Chemistry Assistant enables researchers to extract and analyze MOF synthesis literature without requiring machine-learning expertise.
Takeaways & Limitations
The workflow is limited by token constraints when processing research articles.

Abstract

from arXiv · show

We use prompt engineering to guide ChatGPT in the automation of text mining of metal-organic frameworks (MOFs) synthesis conditions from diverse formats and styles of the scientific literature. This effectively mitigates ChatGPT's tendency to hallucinate information -- an issue that previously made the use of Large Language Models (LLMs) in scientific fields challenging. Our approach involves the development of a workflow implementing three different processes for text mining, programmed by ChatGPT itself. All of them enable parsing, searching, filtering, classification, summarization, and data unification with different tradeoffs between labor, speed, and accuracy. We deploy this system to extract 26,257 distinct synthesis parameters pertaining to approximately 800 MOFs sourced from peer-reviewed research articles. This process incorporates our ChemPrompt Engineering strategy to instruct ChatGPT in text mining, resulting in impressive precision, recall, and F1 scores of 90-99%. Furthermore, with the dataset built by text mining, we constructed a machine-learning model with over 86% accuracy in predicting MOF experimental crystallization outcomes and preliminarily identifying important factors in MOF crystallization. We also developed a reliable data-grounded MOF chatbot to answer questions on chemical reactions and synthesis procedures. Given that the process of using ChatGPT reliably mines and tabulates diverse MOF synthesis information in a unified format, while using only narrative language requiring no coding expertise, we anticipate that our ChatGPT Chemistry Assistant will be very useful across various other chemistry sub-disciplines.

INTRODUCTION

The study presents the ChatGPT Chemistry Assistant as a minimally coded, prompt-engineered system for mining MOF synthesis information from research papers. It evaluates three text-mining processes and uses the resulting synthesis-condition data to support crystallization-outcome prediction and chemistry assistance.

System contributions: The ChatGPT Chemistry Assistant uses verbal instructions and minimal coding to text-mine MOF synthesis conditions and refine the resulting data.The authors frame the approach as generalizable beyond MOF synthesis and assess ChatGPT’s literature-mining intelligence and data-refinement ability.
System contributions: The study introduces a step-by-step ChemPrompt Engineering guide intended to help beginners apply prompt engineering in chemistry.The guide includes chemistry-related examples in the Supporting Information because systematic chemistry prompt-engineering literature is limited.
Text-mining workflow: The three processes trade manual effort, paragraph classification, and embedding-based section filtering to automate synthesis-information extraction at different speeds.Process 2 is fully automated and faster than Process 1, while Process 3 is fastest because embeddings reduce classification tasks.
Text-mining evaluation: More than 26,000 synthesis parameters from approximately 800 compounds were manually compared with ground truth to evaluate all three text-mining processes.The evaluation covered 11 synthesis parameters and assigned each extracted parameter one of three labels.
Text-mining evaluation: All three processes achieved average precision > 95%, recall > 90%, and F1 > 92% across the 11 synthesis parameters.Human evaluation found less than 9.8% NP and 0 cases of hallucination despite irregularities in synthesis reporting.
Crystallization prediction: The extracted synthesis conditions were used to predict whether MOF products form single crystals or crystalline powders.Crystal morphology descriptions in nearly all MOF papers enabled assignment of a crystalline-state target variable for prediction.

Supporting Information

The Supporting Information details general information, ChemPrompt Engineering principles, ChatGPT API text-mining procedures, data-processing workflows, prediction modeling, and creation of a MOF synthesis chatbot.

ChemPrompt Engineering: ChemPrompt Engineering covers hallucination minimization, detailed instructions, structured output, and interactive prompt refinement.These topics are organized in Section S2 and subsections S2.1–S2.4.
Text Mining with ChatGPT API: ChatGPT API text mining includes synthesis-condition summarization, article-section classification, embedding-based filtering, batch processing, and Python-code generation.These procedures are organized in Section S3 and subsections S3.1–S3.5.
ChatGPT-Assisted Chemistry Data Processing Cookbook: The cookbook presents ChatGPT-assisted chemistry data-processing workflows.This material is identified as Section S4.
Prediction Modeling: The supporting information includes prediction modeling for the study’s dataset.This material is identified as Section S5.
Dataset to Dialogue: A MOF synthesis chatbot is created by converting the dataset into dialogue.This material is identified as Section S6.

Section S1. General Information

The study used GPT-3, GPT-3.5/ChatGPT, and GPT-4 alongside ChatGPT-related APIs, while analyzing 228 papers and 225 supporting documents from 32 journals. Prompt refinement, ChatGPT-generated code, and a scikit-learn random forest model supported text mining, preprocessing, chatbot operation, and machine learning.

Language models: Three LLMs—GPT-3, ChatGPT (GPT-3.5), and GPT-4—were involved, each using an autoregressive transformer architecture.The models were developed and maintained by OpenAI, with proprietary training-data and architectural specifics.
Language models: Two APIs—text-embedding-ada-002 and gpt-3.5-turbo—were used, while the GPT-4 API was excluded because access was limited and costs were significantly higher.The gpt-3.5-turbo model was treated as equivalent to the default web-based ChatGPT, so it was called the ChatGPT API.
Source corpus: The source corpus comprised 228 papers and 225 supporting documents from 32 journals, published between May 2004 and March 2023.The documents were obtained with authorization from eight publisher groups and downloaded in PDF format.
Workflow implementation: Prompts were developed through interactive prompt refinement, with further details provided in Supporting Information Section S2.The study’s code and data resources were also documented in Supporting Information and a public GitHub repository.
Workflow implementation: The machine-learning model used the random forest algorithm implemented through the scikit-learn library in Python.Most text-mining, preprocessing, and chatbot-operation code was generated by ChatGPT and tested in a Jupyter notebook.

Section S2. Chemistry ChatGPT Prompt Engineering (ChemPrompt Engineering)

ChemPrompt Engineering uses context grounding, detailed instructions, and structured-output requests to reduce hallucination and organize MOF synthesis information. The examples show that ChatGPT can distinguish relevant synthesis content, handle unknown compounds, and tabulate parameters from supplied context.

S2.1. Principle 1: Minimizing Hallucination: Context-grounded prompting instructs ChatGPT to answer truthfully and use “N/A” when information is missing or uncertain.The examples describe correct handling of unavailable MOF-99999 information and extraction of MOF-419 synthesis conditions from supplied context.
S2.1. Principle 1: Minimizing Hallucination: ChatGPT extracts and summarizes synthesis parameters into tables, including organic linkers, metal sources, solvents, and reaction conditions.The demonstrated outputs organize information for MOF-419 and multiple MOFs from provided context.
S2.2. Principle 2: Implementing Detailed Instructions: Detailed instructions restrict extraction to MOF synthesis and exclude information related to organic linker synthesis.The example shows that explicit relevance criteria help ChatGPT avoid ambiguity in an organic-synthesis discussion.
S2.2. Principle 2: Implementing Detailed Instructions: Specifying fields such as compound name, metal source, amounts, linker, reaction temperature, and reaction time guides table-based summarization.Missing or uncertain values are designated as “N/A.”
S2.2. Principle 2: Implementing Detailed Instructions: Delimiters such as '|' and ':' separate input and output sections, helping ChatGPT process chemical formulas, reactions, and equations.Prompt engineering also guides multi-step tasks within a single conversation.
S2.3. Principle 3: Requesting Structured Output: Structured-output prompts define task instructions and headers for tabulating compound, reagent, solvent, and reaction parameters.The specified headers communicate that the requested operation is tabulation and guide the desired table format.

Section S3. Text Mining with ChatGPT API · S3.2. Classifying Research Article Sections with ChatGPT · S3.3. Filtering Text using OpenAI Embeddings

The ChatGPT API workflow uses chemistry-focused prompts to summarize and classify MOF synthesis information, while OpenAI embeddings filter text by semantic relevance. These processes standardize extraction of synthesis conditions, reaction details, and experimentally comprehensive sections for downstream analysis.

S3.1. Summarizing Synthesis Conditions with ChatGPT: Prompts extract MOF synthesis conditions into tables covering compounds, reactants, amounts, modulators, solvents, temperatures, and reaction times.Missing or uncertain information is represented as “N/A.”
S3.1. Summarizing Synthesis Conditions with ChatGPT: Prompt components guide ChatGPT’s synthesis-condition summaries through chemistry prompt-engineering principles and example input-output tables.The illustrated prompt is a demonstration, and variants can be constructed for specific research needs.
S3.1. Summarizing Synthesis Conditions with ChatGPT: Few-shot prompts extract reaction equipment, vessel, type, product morphology, and compound names, assigning “N/A” when details are absent.Reaction types are determined using prioritized keywords including Diffusion, Microwave, and Conventional.
S3.2. Classifying Research Article Sections with ChatGPT: Few-shot prompts classify synthesis paragraphs by determining whether they contain comprehensive MOF syntheses with explicit reactant quantities or solvent volumes.The classifier returns only “Yes” or “No.”
S3.3. Filtering Text using OpenAI Embeddings: The ada-002 embedding prompt identifies experimental or synthesis sections using compound names, metal sources, organic linkers, amounts, solvents, and solvent volumes.These examples define the semantic content targeted during embedding-based filtering.
S3.3. Filtering Text using OpenAI Embeddings: Embedding-based filtering compares paragraph context with the synthesis prompt using cosine similarity scores ranging from 0 to 1.High similarity is labeled “Yes” and retained, whereas low similarity is labeled “No” and excluded.

S3.4. Batch Text Processing with ChatGPT API

The ChatGPT API enables iterative, concurrent processing of many text inputs through programming constructs such as for loops. This supports efficient batch summarization and classification, with automatically collected responses that can be formatted into tables.

API-based batch processing: The ChatGPT API iterates over multiple text inputs using a programming construct like a “for” loop.This functionality supports repeated question-answering without manual web-interface input.
API-based batch processing: Concurrent processing of many requests enhances efficiency for large-scale text-processing tasks.The API enables a pre-compiled series of inputs to be queried rapidly.
Batch summarization: Batch summarization assembles prompts and context, collects each generated answer, and advances automatically to the next question.String-based responses can be represented in tabular form using “|” and “-” symbols.
Batch classification: Batch synthesis-paragraph classification similarly prepares contextual prompts, collects answers automatically, and directs ChatGPT to respond with “Yes” or “No”.The process uses fixed instructions to classify each paragraph.

S3.5. Generating Python Code with ChatGPT · Paper Parsing

ChatGPT was used to generate Python code that converts research-article PDFs into structured text suitable for prompting and API-based processing. The workflow supports interactive debugging and narrative-language code refinement, including targeted functions and tokenization choices.

Paper Parsing: ChatGPT-generated Python code converts multiple academic PDF files into text strings for incorporation into prompts and processing through the ChatGPT API.This conversion is described as the first step toward autonomous paper parsing and reading.
Paper Parsing: The parsing function divides each page into four sections and records each section’s page and section numbers.Figure S30 describes the function as extracting text from multiple PDF files while preserving page and section identifiers.
Paper Parsing: The requested output dataframe contains file name, page number, section number, content, and token-count columns.The implementation returns the assembled data as a pandas dataframe.
Paper Parsing: ChatGPT code may appear syntactically fluent yet fail during execution, so users can provide error messages for interactive debugging.The paper illustrates this process with a deprecated PyPDF2 method that should be replaced by len(reader.pages).
Paper Parsing: Users can direct implementation changes in narrative language, including requests to use targeted models or functions.This interaction enables code revisions without requiring the user to formulate changes as code.
Paper Parsing: A user request revised token counting to use OpenAI’s tiktoken "cl100k_base" encoding.The revised implementation continues to return a pandas dataframe.
Paper Parsing: Interactive refinement produced fully revised code capable of proficiently parsing academic papers according to specific user requests.Figure S32 presents this refinement as ChatGPT-directed modification of the code.

Text Encoding

Process 3 converts research-paper text segments stored in a DataFrame into OpenAI embeddings, with ChatGPT generating the required functions from documented examples. Embeddings can be saved and reloaded locally, but CSV storage initially causes type inconsistency because embeddings are reloaded as text rather than lists of floats.

Embedding generation: Process 3 requires OpenAI embeddings for each research-paper text segment stored in a DataFrame.ChatGPT can be instructed about the DataFrame contents and given an OpenAI documentation example to generate the embedding code.
Embedding storage: Embeddings are stored locally in CSV files and reloaded into DataFrames to avoid recurring conversions for the same paper pool.The loading function checks whether an 'embedding' column already exists before adding embeddings again.
Embedding storage: Reloaded embeddings initially become strings instead of lists of floats, causing type inconsistency during later use.The issue results from embeddings being stored as text in CSV files; ChatGPT can revise the function after being told the correct embedding type.

Process 3 and Similarity Search

Process 3 performs search and filtering on segmented paper content represented by embeddings. Its functions can be refined to select the 10 most relevant sections for each paper, grouped by unique file name.

Process 3: Process 3 searches and filters segmented paper content using a dataframe containing content and embeddings.The process begins after paper content has been segmented and corresponding embeddings generated.
Similarity Search: 10 sections are selected for each unique file name to remove less relevant content from each paper.This refinement targets the top sections based on relevance within each paper.
Similarity Search: The function is modified to select top neighboring sections separately for different file names.The requested adjustment adds file-name grouping to the similarity-search procedure.

Process 2 and Classification

Process 2 is configured to mirror Process 3, using GPT-4 to generate a Python function that processes dataframe content with a fixed few-shot prompt. The setup also requires standard ChatGPT API-calling code and is depicted in Figure S38.

Process 2 setup: Process 2 mirrors Process 3 through user-provided descriptions of the function name, inputs, and expected outputs for GPT-4.The setup also specifies standard Python code for calling the ChatGPT API from OpenAI’s official documentation.
Process 2 setup: Model_2 accepts a dataframe df with a content column and constructs a prompt by combining a few-shot prompt with each content entry.The implementation instructions explicitly request Python code to build this function.
Process 2 setup: Figure S38 depicts the Process 2 setup using code generated by GPT-4 and refers to Figure S24 for the fixed few-shot prompt.The figure provides a visual summary of the configuration described in the implementation instructions.

Process 1 and Summarization

Process 1 uses either classified synthesis paragraphs from Process 2 or longer human-selected synthesis paragraphs as input. Its implementation applies token-based splitting and prompt-based processing, with the establishment demonstrated using ChatGPT-produced code.

Input sources: Process 1 accepts either Process 2’s classified synthesis-paragraph dataframe or longer human-selected synthesis paragraphs for each paper.Classified segments contain a few hundred tokens, while human-selected paragraphs range from a few hundred to many thousand tokens.
Text processing: Content exceeding 3000 tokens is split at each 3000-token boundary before prompt-based processing.The implementation counts tokens for each content row, prepends contextual text and a tubulation prompt, and then passes the resulting strings onward.
Process establishment: Process 1 establishment is demonstrated with code produced by ChatGPT using GPT-4.The fixed prompt is referenced separately in Figure S22.

Text Parsing

ChatGPT was instructed to generate Python code that converts string-based tabular synthesis data into a readable structured dataframe, then creates sample data to test the function’s accuracy and effectiveness.

Parsing and tabulation: ChatGPT generated a Python function to extract pipe-delimited tabular strings, restructure them, and convert them into a dataframe.The function, named tabulate_condition, accepts a dataframe containing a summarized column with text-formatted tabular data.
Validation: A sample dataframe was generated and used to call and test the function, allowing evaluation of its output accuracy and effectiveness.The testing code was also requested through ChatGPT instructions, as demonstrated in the accompanying figures.

Section S4. ChatGPT-Assisted Chemistry Data Processing Cookbook

Section S4 presents prompt-based workflows for standardizing chemistry data, calculating molecular weights, classifying synthesis outcomes, processing non-English papers, and extracting broader materials properties. ChatGPT accommodates varied formats and contexts, while some linker-standardization cases still require human intervention.

Unit and format unification: Custom prompts standardize reaction times and temperatures across varied units, formats, and equivalent phrases, producing unitless tabulated values.Time conversion selects the longest value when multiple times are present and reports hours without units; temperature conversion outputs °C without units.
Chemical notation standardization: ChatGPT simplifies chemical compound notation standardization, whereas organic-linker normalization to SMILES remains difficult and may require human intervention.The workflow handles diverse formula representations and special characters, but misspellings and inconsistent linker names can remain challenging.
Molecular-weight calculation: ChatGPT-generated Python code calculates molecular weights from standardized compounds and can be extended to batch processing in DataFrames.The procedure excludes undetermined nH2O portions and separates multiple compounds with semicolons before adding molar-mass information.
Synthesis-outcome classification: ChatGPT classifies synthesis descriptions by crystalline state, including colorless crystals, red solids, and microcrystalline powders.The reported literature set contains crystalline MOFs rather than amorphous synthesis products.
Multilingual processing: With minor prompt modifications, ChatGPT extracts and tabulates synthesis parameters from research papers written in languages other than English.This capability is described as challenging for traditional English-based NLP methods.
Broader property extraction: Prompt adaptation extends ChatGPT-assisted literature mining to properties such as BET surface area, sorption behavior, gas uptake, thermal stability, and chemical stability.Adjusting search and summarization modules enables extraction and summarization of BET surface-area data.

Section S5. Prediction Modeling

Section S5 describes curation, descriptor construction, and random-forest modeling for predicting MOF crystalline-state outcomes from synthesis parameters. The workflow reduces feature complexity, encodes chemical and reaction information, and evaluates predictive performance with cross-validation metrics.

Data curation: Synthesis cases containing more than three metals or linkers were pruned to limit one-hot-encoded feature dimensionality.These multicomponent cases were relatively rare and could increase model complexity.
Descriptor construction: Six descriptor sets represented metals, linkers, modulators, solvents, molar ratios, and reaction conditions.Metal descriptors included valency, atomic radius, electron affinity, ionization potential, and electronegativity.
Descriptor construction: Solvents and modulators were assigned to 8 classes using frequency-ranked categories and represented with one-hot encodings.Molecular weights were also included for linker, modulator, and solvent descriptors.
Modeling and evaluation: Random-forest classifiers used an 80% random split, grid-searched tree and leaf-size hyperparameters, and cross-validation with class-weighted accuracy, precision, recall, and F1 score.Models were trained with Scikit-Learn's RandomForestClassifier implementation across varying training sizes.
Outcome labeling: Reaction conditions were classified into 4 categories, while crystalline outcomes were assigned according to descriptions such as single crystals or microcrystallinity.The reaction categories were vapor diffusion, solvothermal, conventional, and microwave-assisted reaction.

Section S6. Dataset to Dialogue: The Creation of a MOF Synthesis Chatbot

The MOF Synthesis Chatbot reformats mined synthesis and publication data into compound information cards, retrieves relevant entries through embedding similarity, and generates context-grounded responses via ChatGPT API prompt engineering. Demonstrations show that it supports synthesis questions, comparative explanations, misinformation handling, and single-crystal synthesis queries.

Data preparation and retrieval: The workflow converts each compound’s synthesis parameters and publication metadata into an information card, then creates embeddings for retrieval.Publication metadata included authors, DOIs, and publication years retrieved from Web of Science.
Data preparation and retrieval: Embedding similarity ranks dataset sections for a user query, and the highest-ranking entry is supplied to ChatGPT’s prompt module for synthesis-focused responses.The chatbot uses the ChatGPT API and bases responses solely on retrieved synthesis information.
Chatbot implementation: The ChatGPT API function combines retrieved search output with a fixed prompt and can access prior conversations to maintain context from the synthesis information card.The authors generated the chatbot code with ChatGPT and describe integration into interactive websites or mobile applications.
Demonstrated capabilities: The assistant compared ZIF-8 and ZTIF-8 by distinguishing metal sources, solvents, co-linkers, and synthesis conditions, while also emphasizing laboratory safety for synthesis attempts.ZTIF-8 uses 5-methyltetrazole as a co-linker, whereas ZIF-8 does not.
Demonstrated capabilities: The demonstrations covered Zirconium MOF procedures, uncertainty about unavailable MOF-499 information, and a literature-supported confirmation that single crystals of MOF-801 can be synthesized.For MOF-499, the assistant requested more information or a specific paper reference rather than providing an unreliable answer.