ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis

INTRODUCTION

• The assistant mines MOF synthesis conditions, evaluates ChatGPT’s text-mining accuracy and data refinement, trains a model to predict crystalline-powder versus single-crystal outcomes, and supports a literature-grounded synthesis chatbot.The authors also present a prompt-engineering guide intended for chemistry tasks and researchers with limited machine-learning or coding experience.Source: INTRODUCTION · pp. 2–3

MATERIALS AND METHODS

• The authors selected 228 papers for diverse synthesis conditions and writing styles, excluding work on post-synthetic modification, MOF catalysis, and composites; both microcrystalline powders and single crystals were eligible.Selection prioritized high-quality, well-cited papers because MOF identity can be ambiguous and some datasets include nonporous inorganic complexes or amorphous coordination polymers.Source: MATERIALS AND METHODS · p. 3, INTRODUCTION · p. 3
• The three workflows respectively summarize preselected experimental text, classify synthesis paragraphs from whole papers, or filter paper segments with embeddings before classification.Process 2 reads sections incrementally and uses few-shot examples to classify synthesis paragraphs; Process 3 retains mid- and high-relevance segments for classification rather than sending embedding matches directly to summarization.Source: MATERIALS AND METHODS · pp. 4, 6, 7
• ChemPrompt Engineering combines hallucination reduction, detailed instructions, and structured output; fixed table headers and explicit directions help constrain extraction and standardize results.Prompts can restrict extraction to MOF synthesis rather than linker synthesis, use “N/A” for unreported parameters, and request multiple rows for variants; interactive refinement adds details and feedback to improve prompts.Source: MATERIALS AND METHODS · pp. 4–5
• ChatGPT-generated Python scripts support batch processing and data unification, while ChatGPT can resolve common chemical abbreviations and standardize units and compound representations; some complex or incomplete linker names still required manual intervention.Embedding filtering cannot reliably bypass classification: synthesis sections mixed with characterization or sorption data may score lower, so retaining mid-relevance segments helps avoid missing them.Source: MATERIALS AND METHODS · pp. 5, 7, 8, 9

RESULTS AND DISCUSSION

• Process 3 was fastest; caching embeddings saved 15–20 seconds (28%–37% faster) on subsequent readings, while Process 1 summarization averaged 13 seconds per paper after 12 hours of selection for 228 papers.Process 2 automates classification and forwarding for summarization, while Process 3 speeds processing by filtering sections before classification.Source: RESULTS AND DISCUSSION · p. 9, MATERIALS AND METHODS · p. 9
• Across 11 parameters, all three processes achieved average precision above 95%, recall above 90%, and F1 above 92%, with no hallucinations detected by human evaluators.The averages weight all 11 parameters equally; evaluators found less than 9.8% NP and zero hallucination cases.Source: RESULTS AND DISCUSSION · pp. 9–10, Figure 5 · p. 10
• Process 2's paragraph-by-paragraph review delivered higher accuracy than Process 3, while Process 3 reduced processing time through filtration and retained acceptable accuracy.All processes identified compound and reagent names well, but struggled with chemical quantities, especially solvent volumes in mixed-solvent systems.Source: RESULTS AND DISCUSSION · pp. 9–10
• The random-forest classifier achieved 87% average accuracy and 92% F1 on the held-out set when predicting single-crystal versus polycrystalline outcomes.Performance was averaged over 10 runs; the model addressed imbalanced classification.Source: RESULTS AND DISCUSSION · p. 11, Figure 6 · p. 12
• Stoichiometric ratios ranked among the most influential crystallization descriptors, followed by reaction time and metal valence, while linker and molecular descriptors also affected growth kinetics.The stoichiometric descriptors included modulator-to-metal, solvent-to-metal, and linker-to-metal ratios.Source: RESULTS AND DISCUSSION · pp. 11–12
• The chatbot retrieves a relevant MOF information card and generates answers from its synthesis data, declining out-of-scope queries to reduce hallucination and retaining context for follow-up questions.The cards include bibliographic information, and the chatbot provides DOI links to original papers for traceability.Source: RESULTS AND DISCUSSION · p. 13
• Prompt changes enabled extraction across languages and information types, with potential application beyond MOFs to fields including organic synthesis, biochemistry, perovskites, and polymers.Demonstrated information types included thermal stability, BET surface area, CO2 uptake, crystal parameters, water stability, and topology.Source: RESULTS AND DISCUSSION · p. 15

CONCLUDING REMARKS

• The dataset supported both predictive modeling of MOF crystallization factors and a data-grounded chatbot, illustrating potential for LLM-assisted chemistry research with limited coding expertise.The authors describe the study as a proof-of-concept for broader scientific applications.Source: CONCLUDING REMARKS · p. 15

• The supporting information is titled “ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF.”Source: Supporting Information · p. 19
• The listed authors are Zhiling Zheng, Oufan Zhang, Christian Borgs, Jennifer T. Chayes, and Omar M. Yaghi.Source: Supporting Information · p. 19

Section S1. General Information

• The study used GPT-3 [1], ChatGPT (GPT-3.5), and GPT-4, distinguishing the default web-based ChatGPT from the GPT-4 chatbot labeled ChatGPT (GPT-4).All three were described as autoregressive transformer models; their training data and architectural specifics were proprietary [2].Source: Section S1. General Information · p. 21

Application Programming Interface (API)

• GPT-3.5 API powered text mining in Processes 1 and 2, while text-embedding-ada-002 powered Process 3; no GPT-4 API was used.As of May 2023, GPT-4 API access required a waitlist and its cost substantially exceeded GPT-3.5 API’s.Source: Application Programming Interface (API) · p. 21

Article Retrieval

• The corpus comprised 228 papers and 225 supporting documents from 32 journals, published between May 2004 and March 2023.The papers were authorized through eight publisher groups; prompts were refined interactively, and most text-mining, preprocessing, and chatbot code was generated by ChatGPT and tested in Jupyter.Source: Article Retrieval · p. 21

Machine Learning Methods

• The study’s random-forest model was implemented with Python’s scikit-learn library.Further details were provided in Supporting Information Section S5.Source: Machine Learning Methods · p. 21

S2.1. Principle 1: Minimizing Hallucination

• GPT-4 correctly said it lacked information about MOF-99999, but still could not describe MOF-419, published after its September 2021 knowledge cutoff.The earlier response recognized that MOF-99999 was unknown but inaccurately described MOF-901.Source: Figure S2 · p. 23, Figure S3 · p. 24, S2.1. Principle 1: Minimizing… · p. 24
• With provided context, ChatGPT identified unavailable synthesis conditions for MOF-99999, extracted MOF-419 synthesis conditions, and organized conditions for MOF-802, MOF-808, DUT-67, and unpublished MOF-521.It extracted conditions for the three MOFs in a single step; MOF-521 was unpublished as of May 2023.Source: Figure S4 · p. 25, Figure S6 · p. 27, Figure S7 · p. 28

S2.2. Principle 2: Implementing Detailed Instructions

• Truthfulness instructions and an explicit scope restriction tell ChatGPT to extract MOF synthesis conditions while ignoring organic-linker synthesis.The prompt also directs it to use “N/A” for missing or uncertain information.Source: S2.2. Principle 2: Implementing… · p. 29
• Prompts can specify synthesis fields for a table, while delimiters and multi-step instructions help organize complex inputs and tasks within one conversation.Requested fields include compound name, metal and linker amounts, temperature, and time.Source: S2.2. Principle 2: Implementing… · p. 30, Figure S10 · p. 31, Figure S11 · p. 32

S2.3. Principle 3: Requesting Structured Output

• Explicit table headers and “N/A” instructions define the requested output, and natural-language directions can change the headers and fields for different sources.The examples include an unpublished manuscript.Source: S2.3. Principle 3: Requesting… · pp. 33, 35, Figure S14 · p. 35
• Few-shot examples help ChatGPT extract challenging parameters such as synthesis vessel type and crystal class.Another prompt requests a table of compound names and reaction-product morphology, using “N/A” when details are absent.Source: Figure S13 · p. 34, S2.3. Principle 3: Requesting… · p. 34

S2.4. Interactive Prompt Refinement

• Interactive evaluation exposed metal sources being misclassified as linkers, prompting a revised instruction to exclude metal sources and inorganic components.The initial prompt specified fixed linker and amount headers and semicolon-separated entries for multiple linkers.Source: Figure S16 · p. 37, S2.4. Interactive Prompt Refinement · pp. 37, 39, Figure S17 · p. 38
• Further refinement prioritized mole over mass units and retained fixed headers and aligned semicolon-separated linker–amount entries; the reported evaluation found the identified issues resolved.The prompt-development process continued testing across scenarios until user satisfaction, with further refinement possible if new issues arose.Source: S2.4. Interactive Prompt Refinement · p. 41, Figure S20 · p. 41, Figure S21 · p. 42

S3.1. Summarizing Synthesis Conditions with ChatGPT

• The synthesis-summary prompt requests a table of compound identity, reagents and amounts, modulator, solvents and volumes, reaction temperature, and time, using “N/A” for missing or uncertain information.The requested fields include metal source and amount, organic linker(s) and amount, and modulator amount or volume.Source: S3.1. Summarizing Synthesis Conditions… · p. 43
• Figure S22 illustrates synthesis-condition prompt components with an example input and output table, referring to three fundamental chemistry prompt-engineering principles.The caption notes that prompt versions can vary with research needs.Source: Figure S22 · p. 43
• A second extraction prompt tabulates each compound’s reaction vessel, reaction type, and product morphology, marking unmentioned information “N/A.”Reaction type is assigned by keyword priority: Diffusion, Microwave, Conventional, then Solvothermal.Source: S3.1. Summarizing Synthesis Conditions… · p. 44
• Few-shot examples demonstrate assigning reaction types and reporting equipment and product appearance; Figure S23 illustrates the prompt strategy with an example input and output table.Examples include solvothermal, conventional, and microwave syntheses.Source: S3.1. Summarizing Synthesis Conditions… · p. 44, Figure S23 · p. 44

S3.2. Classifying Research Article Sections with ChatGPT

• The classification prompt asks whether context contains a comprehensive MOF synthesis with explicit reactant quantities or solvent volumes, requiring a Yes-or-No answer.Source: S3.2. Classifying Research Article… · p. 45
• Figure S24 illustrates few-shot prompts for classifying synthesis paragraphs with two example inputs and their outputs.Source: Figure S24 · p. 45

S3.3. Filtering Text using OpenAI Embeddings

• The embedding prompt targets experimental or synthesis-method sections and specifies compound identity, reagents, amounts, solvent and volume, temperature, and reaction time.Examples include MOF-5, ZrCl4, DMF, 120°C, and 120h.Source: S3.3. Filtering Text using OpenAI… · p. 46
• High cosine similarity to an embedded synthesis prompt retains relevant paragraphs, while low similarity excludes them.Similarity scores range from 0 to 1; multiple prompt versions can be used depending on the study’s requirements.Source: Figure S25 · p. 46

S3.4. Batch Text Processing with ChatGPT API

• A ChatGPT API for-loop can process multiple text inputs automatically, improving efficiency for large-scale tasks compared with manual web-based input.The API can iterate through a precompiled series of inputs and query them rapidly.Source: S3.4. Batch Text Processing with… · p. 47, Figure S27 · p. 48
• For a MOF-5 synthesis-condition question, the API example produces an output that closely aligns with the web-based ChatGPT response.The example uses Python code to submit the question and display the output.Source: Figure S26 · p. 47
• A Python loop can automatically submit preloaded paragraphs for batch summarization and collect each model response.The API responses are strings, so “|” and “-” symbols are used to represent answers in a tabular format.Source: Figure S28 · p. 49
• Batch synthesis-paragraph classification uses preloaded paragraphs and fixed prompts to collect automatic “Yes” or “No” answers.After collecting each answer, the process continues to the next question.Source: Figure S29 · p. 50

Paper Parsing

• ChatGPT-generated Python code converts PDF papers into text strings for use as prompt context and processing through the ChatGPT API.The paper presents this as the first step toward autonomous parsing and reading of academic papers.Source: Paper Parsing · p. 51

S3.5. Generating Python Code with ChatGPT

• The requested parser divides each PDF page into four sections and records file, page, section, content, and token-count data in a dataframe.The prompt specifies section numbers 1–4 and accepts a list of PDF filenames as input.Source: Paper Parsing · p. 51, Figure S30 · p. 51
• When generated code fails during execution, users can provide the error message so ChatGPT can interactively debug it.The figure illustrates revising code after a human copies and pastes an execution error.Source: Paper Parsing · p. 52, Figure S31 · p. 52
• Users can direct code implementation in narrative language, including specifying targeted models or functions.This lets users guide implementation without describing it solely through code.Source: Paper Parsing · p. 53
• A user can request a specific tokenizer by asking ChatGPT to use tiktoken’s “cl100k_base” for token counts.This request is shown as an interactive code revision.Source: Paper Parsing · p. 53, Figure S32 · p. 53
• The illustrated interactive refinement produces a revised parser described as correct and capable of parsing academic papers.Figure S32 shows the fully revised code after user-directed modification.Source: Figure S32 · p. 53

• Process 3 converts each research-paper text segment into an OpenAI embedding by prompting ChatGPT with DataFrame context and an official-documentation code example.Source: Text Encoding · p. 54
• Embeddings can be saved to and reloaded from local CSV files, and the function can skip conversion when the DataFrame already has an embedding column.Source: Text Encoding · p. 55
• CSV reloading can turn embeddings into strings rather than lists of floats; informing ChatGPT of the correct embedding type enables it to revise the function.Source: Text Encoding · p. 56

Process 3 and Similarity Search

• Process 3’s primary objective is searching and filtering paper content, using a dataframe of segmented content and corresponding embeddings.Figure S36 illustrates the configuration, guided by code generated by ChatGPT (GPT-4).Source: Process 3 and Similarity Search · p. 57, Figure S36 · p. 57
• The Model_3 function uses a fixed search prompt and a dataframe with content and embedding columns to perform the search.The instructions ask ChatGPT to convert the fixed prompt into OpenAI embeddings.Source: Process 3 and Similarity Search · p. 57
• To filter less relevant content, the function can be revised to select the top 10 sections for each unique file name.Figure S37 illustrates function optimization according to user requirements, facilitated by ChatGPT (GPT-4).Source: Process 3 and Similarity Search · p. 58, Figure S37 · p. 58

Process 2

• The configuration mirrors Process 3 by specifying the function name, input, and expected output to GPT-4.Source: Process 2 and Classification · p. 59
• The instructions emphasize providing GPT-4 with standard Python code for calling the ChatGPT API, available in OpenAI’s official documentation.Source: Process 2 and Classification · p. 59
• Process 2 uses a ChatGPT-generated Python function, Model_2, that accepts a dataframe and combines a fixed few-shot prompt with its content column.The setup figure depicts Process 2 with code generated by GPT-4.Source: Process 2 and Classification · p. 59, Figure S38 · p. 59

Process 1 and Summarization

• A 3,000-token cutoff segments synthesis content to account for the 4,096-token limit, including the context and fixed prompt.Process 1 accepts either Process 2’s classified synthesis paragraphs, typically a few hundred tokens each, or longer human-selected paragraphs ranging from hundreds to thousands of tokens.Source: Process 1 and Summarization · p. 60
• The Model_1 function counts tokens in each content row, splits content exceeding 3,000 tokens at that boundary, and appends each segment to a fixed tubulation prompt before API submission.The prompt concatenates “Context: ”, the content, and the fixed prompt.Source: Process 1 and Summarization · p. 60

Text Parsing

• ChatGPT generated a Python function that extracts synthesis parameters from API-response table strings and restructures them as a dataframe.The prompt provided an exemplary output from Process 1’s summarized column.Source: Text Parsing · p. 61
• The function merges table data into 11 named columns, removes repeated headers, and drops strings with no table rows or header.It is designed to handle strings containing one or more data rows.Source: Text Parsing · p. 61
• A sample dataframe was used to call and test the generated function, allowing evaluation of its output and accuracy.ChatGPT generated code to create the sample dataframe from the exemplary input.Source: Text Parsing · p. 62

Reaction Time and Temperature Conversion

• GPT-4 prompts standardize reaction times and temperatures across formats, handling semantic variants, malformed units, blanks, and 50–100 data points per operation.The prompts treat “overnight” as 12 hours, use the longest time when multiple values appear, and convert room temperature to 25 °C while selecting the highest temperature in ramping instructions.Source: Reaction Time and Temperature Conversion · pp. 63–64

Chemical Formula Conversion and Molecular Weight Calculation

• ChatGPT standardizes varied compound names into formula notation, then generates Python code to calculate molecular weights and batch-process DataFrames.The reported workflow executes the generated code on standard notations and then modifies it into a function for batch calculations.Source: Chemical Formula Conversion and… · pp. 65–67

Reaction Outcome Classification

• Synthesis outcomes are classified as single-crystal (SC) when authors clearly report crystals, or polycrystalline (P) for solids, powders, or descriptions not specifying single crystals; the selected literature contained no amorphous outcomes.Source: Reaction Outcome Classification · p. 68

Unifying Linker Representations

• For linker unification, abbreviations are expanded, then the PubChem API retrieves CID and SMILES codes to merge synonyms; unresolved entries require manual review, and direct ChatGPT SMILES generation risks inaccuracy.The workflow builds a dictionary of names sharing a PubChem ID and SMILES code, then replaces those names and abbreviations with SMILES codes.Source: Unifying Linker Representations · pp. 69–70

Linguistic Versatility in Synthesis Condition Summarization

• With minor prompt modifications, ChatGPT extracted and tabulated synthesis parameters in English from German, Chinese, Japanese, and Korean papers, a task challenging for traditional English-based NLP methods.The demonstrations include papers in all four languages.Source: Linguistic Versatility in Synthesis… · p. 76, Figure S55 · p. 76, Figure S56 · p. 77

Multidimensional Physicochemical Information Summarization

• By modifying search and summary prompts, ChatGPT extracts physicochemical and structural properties; reloaded paper embeddings support new-target searches without reprocessing, including TGA decomposition temperatures and BET surface area or pore volume.The same approach can target sorption behavior, gas uptake, thermal or chemical stability, topology, and crystal parameters.Source: Multidimensional Physicochemical… · pp. 80, 83

Prediction Modeling

• 764 curated samples from Process 1 were used for training after cases with more than three metals or linkers were dropped.The authors selected Process 1 because its mined conditions had the fewest errors that could affect modeling.Source: Section S5. Prediction Modeling · p. 85
• Descriptors encoded metal properties, linker motifs and molecular features, component molar ratios, solvent and modulator classes, and reaction conditions.Solvents and modulators were grouped into eight one-hot encoded classes; mixtures used molar-weighted component descriptors or normalized molar fractions, with zeros for absent species.Source: Section S5. Prediction Modeling · pp. 85, 87
• Random-forest classifiers used an 80% random split, grid-searched tree-estimator and leaf-size settings, and cross-validation to assess class-weighted accuracy, precision, recall, and F1.Feature permutation importance was measured as the percent decrease in accuracy when one descriptor at a time was permuted.Source: Section S5. Prediction Modeling · p. 85
• Reaction conditions were classified as vapor diffusion, solvothermal, conventional, or microwave-assisted, while outcomes were labeled single-crystal or polycrystalline from product descriptions.Descriptions mentioning crystals were classified as single-crystal; microcrystalline, powder, solid, or unspecified morphology were classified as polycrystalline.Source: Section S5. Prediction Modeling · p. 87
• Recursive feature elimination with 5-fold cross-validation reduced the initial 70 descriptors to 26, and permutation analysis assessed the selected features’ influence on accuracy.The descriptor set included chemical, molecular, ratio, solvent-type, and modulator-type features.Source: Section S5. Prediction Modeling · pp. 85, 87

• The chatbot retrieves relevant embedded information cards by query similarity, then uses a ChatGPT API prompt to answer from the supplied synthesis information.Cards combine compound synthesis parameters with publication data; the system can return top-3 or top-5 matches within the 4096-token gpt-3.5-turbo budget and can use conversation history for context.Source: Section S6. Dataset to Dialogue: The… · pp. 92–94
• A proposed web or mobile interface would send user queries to the server and return responses informed by conversation history.The paper describes this as a potential application of the chatbot function.Source: Section S6. Dataset to Dialogue: The… · p. 96
• For ZIF-8 and ZTIF-8, the assistant compared synthesis characteristics and gave a step-by-step ZIF-8 procedure with safety precautions.It also advised the user to learn essential lab techniques before attempting synthesis.Source: Section S6. Dataset to Dialogue: The… · p. 97
• For zirconium MOFs, the assistant described two synthesis procedures and explained how modulator choice can affect crystal growth, yield, structure, and properties.The example discussed acetic acid and possible outcomes if it was omitted.Source: Section S6. Dataset to Dialogue: The… · p. 97
• After failing to find MOF-499, the assistant responded to the corrected MOF-419 query with a synthesis procedure and described common post-synthesis steps, while noting their specifics were not in the paper.Source: Section S6. Dataset to Dialogue: The… · p. 97
• For MOF-801, the assistant discussed single-crystal synthesis, its value for structural characterization, differences from powder synthesis, and other zirconium MOFs in the paper.Source: Section S6. Dataset to Dialogue: The… · p. 97
• The assistant provided contextually aware, focused responses that helped researchers navigate MOF synthesis and access relevant data.The paper presents this as a useful tool for chemistry researchers.Source: Section S6. Dataset to Dialogue: The… · p. 97