PaperlayerView exact revision on arXiv

Source-linked AI summary

14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon

Kevin Maik Jablonka, Qianxiang Ai, Alexander Al-Feghali, Shruti Badhwar, Joshua D. Bocarsly, Andres M Bran, Stefan Bringuier, L. Catherine Brinson, Kamal Choudhary, Defne Circi, Sam Cox, Wibe A. de Jong, Matthew L. Evans, Nicolas Gastellu, Jerome Genzling, María Victoria Gil, Ankur K. Gupta, Zhi Hong, Alishba Imran, Sabine Kruschwitz, Anne Labarre, Jakub Lála, Tao Liu, Steven Ma, Sauradeep Majumdar, Garrett W. Merz, Nicolas Moitessier, Elias Moubarak, Beatriz Mouriño, Brenden Pelkie, Michael Pieler, Mayk Caldas Ramos, Bojana Ranković, Samuel G. Rodriques, Jacob N. Sanders, Philippe Schwaller, Marcus Schwarting, Jiale Shi, Berend Smit, Ben E. Smith, Joren Van Herck, Christoph Völker, Logan Ward, Sean Warren, Benjamin Weiser, Sylvester Zhang, Xiaoqi Zhang, Ghezal Ahmad Zia, Aristana Scourtas, KJ Schmidt, Ian Foster, Andrew D. White, Ben Blaiszik

arXiv:2306.06283v4cond-mat.mtrl-scics.LGphysics.chem-ph

TL;DR

Traditional laboratory interfaces can struggle to capture rich, interconnected experimental data, motivating investigation of LLMs as conversational interfaces. This paper chronicles a hackathon exploring LLM applications and reports that working prototypes were produced in less than two days.

Abstract

from arXiv · show

Large-language models (LLMs) such as GPT-4 caught the interest of many scientists. Recent studies suggested that these models could be useful in chemistry and materials science. To explore these possibilities, we organized a hackathon. This article chronicles the projects built as part of this hackathon. Participants employed LLMs for various applications, including predicting properties of molecules and materials, designing novel interfaces for tools, extracting knowledge from unstructured data, and developing new educational applications. The diverse topics and the fact that working prototypes could be generated in less than two days highlight that LLMs will profoundly impact the future of our fields. The rich collection of ideas and projects also indicates that the applications of LLMs are not limited to materials science and chemistry but offer potential benefits to a wide range of scientific disciplines.

6Laboratory of Artificial Chemical Intelligence (LIAC), Institut des Sciences et Ing´enierie Chimiques, … I. INTRODUCTION

The introduction frames LLMs as a potential way to address the mismatch between rigid scientific tools and context-dependent chemistry workflows, and presents hackathon prototypes spanning four application areas. It also describes natural language as a means to connect diverse tools and make knowledge more accessible, alongside opportunities for education.

I. INTRODUCTION

  • Scientific tools require rigid, often incompatible formats, while context-dependent procedures resist tabular conversion without information loss, forcing application-specific glue code.Source: §I · p. 4
  • The paper presents LLMs as a way to address these challenges through flexible interfaces, task-solving, natural-language programming, and structured information extraction.Source: §I · pp. 4–5
  • Across wet-lab, computational, interface, and classroom applications, hackathon prototypes were built in hours, though they were not polished products, and the authors say they would otherwise take months.Source: §I · p. 5
  • The projects fall into predictive modeling, automation and novel interfaces, knowledge extraction, and education, including classification and regression and modeling with established or fuzzy-context concepts.Source: §I · p. 5
  • The projects suggest natural language could serve as “glue” connecting tools across existing formats and standards.They propose using natural-language descriptions to bridge the diversity of tools rather than focusing on new formats or standards.Source: §I · p. 6
  • LLMs can make knowledge more accessible by extracting structured information from unstructured text.This is illustrated by projects in the knowledge-extraction category.Source: §I · p. 6
  • The education-category project shows that LLMs can offer new educational opportunities.Source: §I · p. 6

A. Predictive modeling

This section presents predictive modeling as an extension of earlier LIFT-based property prediction and ICL-guided experimentation. The hackathon projects extend those approaches and draw on established quantum-machine techniques.

A. Predictive modeling

  • Jablonka et al. used LIFT to predict solubility and HOMO-LUMO gaps from SELFIES or SMILES molecular representations.Source: A. Predictive modeling · pp. 6–7
  • Ramos et al. used LIFT with in-context learning for Bayesian optimization to guide experiments without training models.Source: A. Predictive modeling · p. 7
  • The projects extend earlier predictive-modeling results in novel ways and leverage established techniques from quantum machine learning.Source: A. Predictive modeling · p. 7
  • Here, LIFT also covers in-context learning setups that convert structured datasets into text prompts for an LLM.Source: A. Predictive modeling · p. 7

a. Molecular Energy Predictions · b. Text2Concrete

The Molecular Energy Predictions project used LIFT to predict molecular atomization energies from string representations, while Text2Concrete explored LLM predictions of concrete strength. Both examples show how LLMs can use domain-specific information, including natural-language context, to support materials and chemistry modeling.

a. Molecular Energy Predictions

  • LIFT models using simple line representations such as SMILES and SELFIES predicted molecular atomization energies on QM9-G4MP2 with R2 > 0.95 on a holdout test set.Models were tuned on 90% (117,232 molecules) and tested on 10% (13,026 molecules) of the QM9-G4MP2 dataset.Source: a. Molecular Energy Predictions · pp. 7–8
  • A Δ-ML approach tuned to predict the difference between G4(MP2) and B3LYP showed good agreement and can reach chemical accuracy.The table caption describes the approach as reaching chemical accuracy and reports baseline MAEs of 0.0223 eV for FCHL G4(MP2) prediction, and 0.0045 eV (SchNet) and 0.0052 eV (FCHL) for Δ-ML.Source: a. Molecular Energy Predictions · pp. 7–8
  • Fine-tuning GPT-2 with LoRA produced comparable results on consumer hardware, making LIFT more accessible without relying on the OpenAI API.Source: a. Molecular Energy Predictions · p. 8

b. Text2Concrete

  • The largest LLMs could predict concrete compressive strength without fine-tuning by learning from examples supplied in the prompt.This few-shot approach performed comparably to Gaussian process regression but less accurately than random forest in the case study.Source: b. Text2Concrete · pp. 8–9
  • Natural-language design context reduced outliers and enabled Text2Concrete to outperform the random forest model (R2 of 0.67 and 0.72, respectively).The context encoded a design principle linking higher water-to-cement ratios with reduced strength.Source: b. Text2Concrete · p. 9
  • Text2Concrete illustrates how LLMs can incorporate “fuzzy” domain expertise that is difficult to capture in conventional modeling workflows.The authors identify testing how such context alterations affect performance as future work and suggest that natural-language hypotheses could be assessed by their impact on model accuracy.Source: b. Text2Concrete · p. 9

c. Molecule Discovery by Context · d. Text template paraphrasing

These sections describe using contextual scientific-text embeddings to rank candidate hydrogen carriers and paraphrasing LIFT prompt templates to augment training data and reduce template overfitting.

c. Molecule Discovery by Context

  • Tshitoyan et al. used a Word2Vec approach to embed words into vector space; for materials-science abstracts, compound embeddings such as Li2CuSb could support materials discovery by measuring their distance (cosine similarity) to concepts.Word2Vec predicts, for a word, the probability of all possible next words in a vocabulary; embeddings capture syntactic and semantic details of lexical items.Source: c. Molecule Discovery by Context · p. 9
  • Unlike static Word2Vec embeddings, LLM embeddings are contextualized to the sentence in which words occur, capturing corpus-specific context.This contextual embedding approach motivated testing whether similar representations could support hydrogen-carrier discovery.Source: c. Molecule Discovery by Context · pp. 9–10
  • With ten training examples and natural-language design context, LIFT in-context learning outperformed random-forest and Gaussian-process-regression baselines for concrete compressive strength.The context can encode fuzzy design rules that are difficult to incorporate into conventional machine-learning models.Source: c. Molecule Discovery by Context · p. 10
  • The GlobusLabs team used ScholarBert sentence embeddings as molecular fingerprints, then ranked candidates by similarity to known hydrogen carriers.They averaged embeddings of sentences found in a scientific-article corpus for each candidate molecule.Source: c. Molecule Discovery by Context · p. 10
  • Visual inspection indicated that the selected candidate molecules shared similarities with known hydrogen-carrier molecules.The approach targets hydrogen carriers relevant to energy storage.Source: c. Molecule Discovery by Context · p. 10

d. Text template paraphrasing

  • In low-data settings, fixed LIFT prompt templates use examples to teach syntax; in big-data settings, repeated templates may erode general language abilities.Text augmentation is challenging because discrete text must remain syntactically and semantically valid.Source: d. Text template paraphrasing · pp. 10–11
  • LLMs can paraphrase prompt templates to generate high-quality alternatives for LIFT training, augmenting data and reducing overfitting to one template.The passage notes this approach has also been explored by Dai et al..Source: d. Text template paraphrasing · p. 11

e. Genetic algorithm using an LLM · B. Automation and novel interfaces

The McGill team found that GPT-3.5 could fragment and recombine SMILES molecules into chemically reasonable structures, and explored LLM-guided molecular optimization. The section also describes tool-using LLMs as a way to connect users with external tools through natural language.

e. Genetic algorithm using an LLM

  • GPT-3.5 fragmented molecules at rotatable bonds with a 70 % success rate, without fine-tuning.The authors took this as evidence that GPT-3.5 understands SMILES and aspects of their relation to chemical structures.Source: e. Genetic algorithm using an LLM · p. 11
  • Chemists judged all 32 LLM-generated recombinations chemically reasonable, compared with 21/32 random recombinations.The LLM often combined fragments from both parent molecules into reasonable structures.Source: e. Genetic algorithm using an LLM · pp. 11–12
  • With 30 parent molecules and their Tanimoto-similarity-to-vitamin-C scores, the LLM proposed new molecules; visual inspection suggested chemically reasonable modifications.The authors said future work should systematically investigate potential improvements.Source: e. Genetic algorithm using an LLM · p. 12
  • When suitably conditioned, LLMs might generate new chemically sensible structures, not just reproduce known ones.The approach still faces difficulties producing valid SMILES without explicit fine-tuning, and context length limits how many parent molecules can be supplied.Source: e. Genetic algorithm using an LLM · pp. 12–13

B. Automation and novel interfaces

  • LLMs can act as agents that autonomously use external tools; giving them tool access and requiring step-by-step thought can shift hyperconfident models that often hallucinate into systems reasoning from observations made by querying robust tools.Source: B. Automation and novel interfaces · p. 13
  • Natural-language interfaces can make external tools more accessible without requiring users to learn tool-specific APIs and more interoperable without glue code.The section also notes that tool combinations may enable systems to do things that their individual parts could not do before.Source: B. Automation and novel interfaces · p. 13
  • Bran et al. used natural-language prompts for assistants combining Google search and IBM RXN, while Boiko et al. connected LLMs to laboratories through cloud-lab APIs.The latter system could use external tools to plan a synthesis and execute it in a cloud lab.Source: B. Automation and novel interfaces · p. 13

a. MAPI-LLM · b. sMolTalk

MAPI-LLM turns text prompts into Materials Project workflows and uses in-context examples when property data are unavailable; sMolTalk generates visualization code from natural-language instructions, though robustness remains a challenge.

a. MAPI-LLM

  • MAPI-LLM converts text prompts into Materials Project API queries for material-property questions.The system takes an initial step toward making these queries accessible through natural-language input.Source: a. MAPI-LLM · p. 14
  • It supports classification and regression questions and can build a two-step workflow that checks metallicity before calculating a band gap.For example, it can answer whether Fe2O3 is magnetic, or condition a band-gap calculation for Mn23FeO32 on whether it is metallic.Source: a. MAPI-LLM · p. 14
  • When Materials Project data are unavailable, MAPI-LLM uses in-context learning with similar-material data to infer the unknown property.This approach bridges data gaps and is described as enhancing the system’s robustness and versatility.Source: a. MAPI-LLM · p. 14

b. sMolTalk

  • sMolTalk uses few-shot prompting to generate 3dmol.js code for retrieving protein structures and creating custom visualizations from natural-language requests.The prototype supports commands such as coloring parts of a structure.Source: b. sMolTalk · p. 15
  • Users can phrase visualization requests in varied ways, and the LLM translates them into instructions the software can interpret.Examples include “color” or “colour” and descriptions such as “light yellow” or “pale yellow.”Source: b. sMolTalk · p. 15
  • sMolTalk remains vulnerable to prompt fragments leaking into outputs, motivating robustness improvements such as retries, error feedback, prompt engineering, or documentation-based knowledge.These are proposed approaches for addressing the challenge, not reported fixes.Source: b. sMolTalk · p. 15
  • LLM interfaces can also be used with VMD and extended with speech-to-text such as Whisper for voice control.The work demonstrating an LLM interface for software is attributed to Glenn Hocky and Andrew White.Source: b. sMolTalk · p. 15
  • A chat interface for PyMOL could connect protein-engineering tools and lower biologists’ barrier to using in-silico methods such as RosettaFold and RFDiffusion.This is presented as a potential application of the LLM-agent approach.Source: b. sMolTalk · p. 16

c. ELN interface: whinchat · d. BOLLaMa: facilitating Bayesian optimization with large language models

Whinchat and BOLLaMa use conversational LLM interfaces to make experimental data and Bayesian optimization more accessible to chemists. Together, the projects illustrate how such interfaces can support data interpretation, experiment planning, and more efficient workflows.

c. ELN interface: whinchat

  • Whinchat embeds an LLM chat assistant in datalab, attaches it to a sample, and uses structured and unstructured data from the system’s JSON API to offer a flexible interface to experimental records.This approach addresses the difficulty traditional interfaces have capturing rich, interconnected experimental data.Source: c. ELN interface: whinchat · p. 17
  • Whinchat can contextualize linked experiments, resolve acronyms and shorthand, summarize complex entries, and render data as flowcharts or Markdown tables.The project also describes tailoring the assistant with a group’s knowledge base and communication history as a possible extension.Source: c. ELN interface: whinchat · p. 17
  • Whinchat can suggest materials, synthesis protocols, and characterization techniques; examples include hints about which NMR-active nuclei to probe in a sample.Its small context window limits how much JSON data can be included, restricting conversations to relatively few samples; embeddings or an agent that queries the ELN API are proposed ways to address this.Source: c. ELN interface: whinchat · p. 17

d. BOLLaMa: facilitating Bayesian optimization with large language models

  • Existing frameworks require significant background knowledge and coding experience not conventionally taught in chemistry curricula, so many chemists cannot benefit from BO; BOLLaMa showed LLMs can lower that barrier with a natural-language, chat-like interface to BO algorithms.Source: d. BOLLaMa: facilitating Bayesian… · p. 18
  • Connecting BOLLaMa to data management could let users persist experimental results and use prior records to bootstrap optimization runs.The project presents this as a potential synergy with an LLM interface to a data-management system.Source: d. BOLLaMa: facilitating Bayesian… · p. 18
  • The projects illustrate how LLM interfaces or automation may enhance chemistry and materials-science workflows, streamline processes, reduce human error, and increase productivity.The paper describes replacing “glue code” with natural language and learning software libraries through chat as possible routes.Source: d. BOLLaMa: facilitating Bayesian… · p. 19

C. Knowledge Extraction · a. InsightGraph · b. Extracting Structured Data from Free-form Organic Synthesis Text

The section describes LLMs for extracting and structuring knowledge from chemical literature, including knowledge graphs for materials and structured data from organic synthesis procedures.

C. Knowledge Extraction

  • LLMs can mine chemical literature, while PaperQA retrieves, summarizes, and contextualizes relevant passages to answer questions with citations from libraries of hundreds of documents.Source: C. Knowledge Extraction · p. 19

a. InsightGraph

  • Dunn et al. and Walker et al. used expert-corrected zero-shot outputs to fine-tune LLMs for JSON extraction, improving task performance.Source: a. InsightGraph · p. 20
  • With an example JSON, a high-level schema, and candidate entities and relationships, GPT-3.5-turbo generated a knowledge graph from text on polymer nanocomposite properties and composition.The graph representation could be visualized using tools such as neo4j.Source: a. InsightGraph · p. 20, C. Knowledge Extraction · p. 20
  • An optimized InsightGraph could help compare material types and uses across articles, support literature reviews, and create materials-specific datasets for question answering and machine learning.These are proposed potential applications of the prototype.Source: a. InsightGraph · p. 20

b. Extracting Structured Data from Free-form Organic Synthesis Text

  • Manual conversion of the large organic-synthesis literature into structured data is unrealistic, although such data are needed to build reaction-prediction and condition-recommendation models.Source: b. Extracting Structured Data from… · pp. 20–21
  • 93 % of the components were extracted from free-text reaction descriptions into valid JSON after fine-tuning text-davinci-003 on only 300 prompt-completion pairs.The resulting structured reaction data could increase training data for models predicting reaction conditions and yields.Source: b. Extracting Structured Data from… · p. 21
  • Parameter-efficient fine-tuning of Alpaca with LoRA did not yield a model capable of extracting structured data from organic synthesis procedures [85].Source: b. Extracting Structured Data from… · p. 21

c. TableToJson: Structured information from tables in scientific papers · d. AbstractToTitle & TitleToAbstract: text summarization and text generation

The projects used LLMs to convert scientific tables into structured JSON and to generate titles or abstracts from abstracts or titles. Results show strong extraction and text-generation performance, alongside schema, formatting, and authorship limitations.

c. TableToJson: Structured information from tables in scientific papers

  • TableToJson targets table data on properties, synthesis, and experiments, converting it to fixed-schema records for analysis, computational workflows, and databases.The team reported that both approaches could generate JSON, while standard prompting did not guarantee schema compliance.Source: c. TableToJson: Structured information… · pp. 22–23
  • 100% of generated values were error-free across the studied tables, although the model sometimes failed to follow the requested schema.The OpenAI model extracted cell information into expected schema locations and returned valid JSON objects.Source: c. TableToJson: Structured information… · p. 22
  • Jsonformer reads schema keys and generates only value tokens, guaranteeing syntactically valid JSON, unlike unconstrained LLM output.Special characters may still cause problems, which careful prompting can often address.Source: c. TableToJson: Structured information… · p. 22
  • Jsonformer produced 100% correct values in most cases, but accuracy fell to 80% for scientific-notation errors and 46% for complex table content.Adding specific prompt explanations raised accuracy to 100% in most cases.Source: c. TableToJson: Structured information… · pp. 22–23

d. AbstractToTitle & TitleToAbstract: text summarization and text generation

  • AbstractToTitle and TitleToAbstract were introduced in JARVIS-ChemNLP [95], with train/test splits and a leaderboard for comparing LLMs.The tasks generate a title from an abstract and an abstract from a title.Source: d. AbstractToTitle & TitleToAbstract:… · p. 24
  • On condensed-matter physics (cond-mat) data, fine-tuning the model improved performance: its ROUGE-1 score was 39.0%, compared with 30.8% for an untrained model.Source: d. AbstractToTitle & TitleToAbstract:… · p. 24
  • Fine-tuned GPT-2-medium achieved a 31.7% ROUGE score, a starting point for pre-suggestion text applications.The model was fine-tuned for generating abstracts from titles.Source: d. AbstractToTitle & TitleToAbstract:… · p. 24
  • Title-to-abstract generation could extend to full-length drafts with tables, figures, and results, but LLMs should assist rather than author manuscripts.The paper notes recent guidelines for using LLM-generated text in technical manuscripts.Source: d. AbstractToTitle & TitleToAbstract:… · p. 24

D. Education · II. CONCLUSION

The Education section presents I-Digest as a way to turn lecture recordings into digital tutoring, while the conclusion describes rapid prototyping and identifies opportunities and challenges for LLM use in science.

D. Education

  • I-Digest uses Whisper transcripts to create a digital tutor with an effectively infinite supply of questions, unlike conventional materials, and could improve through student feedback.Figure 10 describes generating questions and answers from lecture transcripts and detecting chemical mentions to link further information, including PubChem.Source: D. Education · p. 25
  • I-Digest could use pre-video questions to help students skip familiar material or post-video questions to direct them to relevant timestamps, notes, books, or exercises.The authors also suggest extending the approach to lecture notes or books.Source: D. Education · p. 25

II. CONCLUSION

  • Hackathon teams built prototypes for complex tasks in hours rather than the months such programming might previously have required, and reported strong results in the time-constrained setting.The authors attribute the teams’ energy and motivation partly to the low cost of failure.Source: §II · p. 26
  • The LIFT framework makes conventional approaches more accessible, while LLMs can model chemistry and materials science using context such as fuzzy design rules or unstructured data.The conclusion also notes that these approaches can reuse established concepts such as ∆-ML.Source: §II · p. 26
  • LLMs can turn fuzzy abstractions into code and enable end-user customization; English-based tool logic yields shorter, easier-to-understand code and suggests natural-language interoperability.The authors also report fewer dependencies in the resulting code.Source: §II · p. 26
  • The projects relied on general reasoning and supplied chemistry through context or fine-tuning, not on LLM chemistry knowledge; their OpenAI API models were opaque and lacked guaranteed continued access.The authors note that open-source models were generally harder to use than the API.Source: §II · p. 26
  • Fragile zero- and few-shot performance and the lack of frameworks for evaluating context-, unstructured-data-, or tool-using models motivate new benchmarks, automated tests, and version tracking for reproducibility.Reproducibility requires recording the training data, LLM, external tools, and prompting setup.Source: §II · p. 27
  • The authors call for safe-use practices, evaluation and sharing standards, robust deployments, and chemists and materials scientists who can use LLMs efficiently while critically assessing outputs.They argue that leveraging LLMs requires a community effort spanning chemists, computer scientists, lawyers, philosophers, and ethicists.Source: §II · p. 27

Supporting Information for: 14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model

The supporting information begins with the paper’s contributor list and institutional affiliations.

Supporting Information

  • The front matter lists the paper’s contributors and their institutional affiliations.The affiliations include EPFL, MIT, McGill University, Duke University, and the University of Rochester.Source: Supporting Information for: 14… · p. 38

I. Predictive Modeling … 2. Solution

This section reports LLM applications in molecular-energy and concrete-strength prediction, hydrogen-carrier screening from scientific text, and paraphrasing chemical-data templates. It also presents templated responses about molecules and target properties, noting that the sentence does not use SMILES and the expected answer lacks curly brackets.

A. Leveraging LLMs for Accurate Molecular Energy Predictions

  • The GPT-3 LIFT model reached chemical accuracy (< 1.0 kcal/mol) for ∆-ML atomization energies, while direct SMILES fine-tuning had errors an order of magnitude above baselines.Models were fine-tuned on molecular SMILES or SELFIES strings to predict G4(MP2) energies or the difference between B3LYP and G4(MP2) energies. The study reports that SMILES was marginally better than SELFIES; Table I covers QM9-G4MP2 results, with baseline MAE versus this work’s MAD.Source: A. Leveraging LLMs for Accurate… · pp. 40–41, Table I · p. 40, One sentence summaries · p. 42
  • LoRA tuning froze GPT-2’s weights and trained adapter layers, but the 124-million-parameter model performed slightly worse than GPT-3 on G4MP2 and moderately worse on the energy-difference task.The authors aimed to develop a free, open-source framework and identified testing other LLMs and molecular representations that better capture 3D environments as future work.Source: A. Leveraging LLMs for Accurate… · p. 41

B. From Text to Cement: Developing Sustainable Concretes Using In-Context Learning

  • With qualitative concrete-design context, ICL reached R-squared = 0.71, surpassing RF; small-data scores were ICL 0.5, GPR 0.54, and RF 0.67.Incorrectly changing the stated fly-ash-to-GGBFS ratio lowered ICL’s R-squared to 0.6, showing that context quality can affect prediction; the impact may depend on the rule. The evaluation used 240 formulations and repeated random training and test sampling.Source: B. From Text to Cement: Developing… · pp. 43–44, Figure 2 · p. 43

C. Molecule Discovery by Context

  • The LLM screened around 100 molecules per second versus around 30 seconds per molecule for quantum chemistry, a 3000 times speedup; its recommendations performed better than random.ScholarBERT ranked candidates by cosine similarity between molecule representations derived from literature contexts; it performed better on structurally similar candidates than on random candidates, with a tendency to favor five- and six-member rings, including unexpected halogens.Source: C. Molecule Discovery by Context · p. 45, One sentence summaries · p. 46

1. Problem

  • Scientific text augmentation is difficult because text is discrete, and traditional approaches can require inefficient manual checks, especially for chemistry applications.Source: §1 · p. 47

2. Solution

  • The chemistry NLP project used LLM prompts to paraphrase LIFT data templates while retaining their chemical-data placeholders; example paraphrases are shown for GPT-4 and Claude.The prompt specified preserving the original meaning and every placeholder in each rephrased sentence.Source: §2 · pp. 47–48
  • The response templates use placeholders for SMILES descriptions, target names, values, and units in several phrasings.Source: §2 · p. 48
  • The example notes that the SMILES representation is unused in the sentence and that the expected answer has no curly brackets.Source: §2 · p. 48

3. Impact … B. sMolTalk

The projects explored LLMs for training-data augmentation, molecular optimization, materials-science questions, and molecular visualization. Their capabilities varied by task, with limitations including unreliable outputs, prompt leakage, and the need for systematic validation.

3. Impact

  • The LIFT project uses LLM paraphrasing to generate high-quality prompt templates efficiently, but the setup worked with GPT-4 and Claude v1 rather than less capable open-source models.The authors propose that future open-source models could enable broader, more cost-effective use; follow-up ablations are needed to test effects on regression and classification.Source: §3 · p. 49, §4 · p. 49, One sentence summaries · p. 49

One sentence summaries

  • The one-sentence summaries frame the projects as template rephrasing for LLM training, LLM-proposed genetic-algorithm children, and tool-using agents for materials questions.The summaries identify data augmentation, potentially more efficient genetic operations, and an accessible interface to materials-informatics tools as their respective aims or impacts.Source: One sentence summaries · pp. 49, 50, 51, 55

E. GA without genes

  • GPT-3.5-turbo often fragmented molecules into valid SMILES and, when prompted to fragment before mixing, sometimes produced children resembling both parents and closer in size to them.Fragments were not in the original molecule for 2/10 cases, and valid SMILES could not be produced for 1/10 even after ten attempts; invalid children were often produced when mixing.Source: E. GA without genes · p. 50, Figure 5 · p. 52, Figure 6 · p. 53
  • For vitamin C similarity optimization, GPT-3.5-turbo was prompted iteratively with scored SMILES, and the LLM-assisted search found multiple new best molecules over 12 generations.The procedure used 30 top-scoring examples, requested 25 new SMILES per iteration for 20 iterations, and used temperatures typically below 0.1; the figure shows multiple new bests but does not report their scores in the supplied passage.Source: E. GA without genes · p. 50, Figure 7 · p. 54

A. Using chain-of-thought and chemical tools to answer materials questions

  • MAPI-LLM combines natural-language prompts, iterative tool selection, and in-context learning to answer materials questions and suggest synthesis or decomposition reactions.It can use tools such as the Materials Project dataset and reaction-network package, and can draw on other tools when a material is not found in the dataset.Source: A. Using chain-of-thought and chemical… · p. 55

One sentence summaries

B. sMolTalk

  • sMolTalk uses few-shot prompts to generate JavaScript code for 3dmol.js, enabling chemists to control a molecular visualization through natural-language commands.The prompt is updated with the viewer’s current state before each subsequent command.Source: B. sMolTalk · p. 57, Figure 9 · p. 57
  • sMolTalk’s potential for structural reasoning depends on additional specialized tools, while prompt leakage and unreliable recall of PDB IDs or CAS numbers limit the current approach.The authors describe API-based retrieval and a ReAct agent as ways to improve reliability, and suggest fine-tuning as a possible route to better code generation.Source: B. sMolTalk · pp. 57–58

One sentence summaries … A. InsightGraph

The projects use LLMs to make scientific tools and experimental data more accessible through natural-language interfaces, while also exploring automated knowledge-graph extraction. Their prototypes show useful capabilities, alongside robustness, context-size, and evaluation limitations.

One sentence summaries

  • Without consulting documentation, an LLM often generated 3dmol.js commands that performed requested actions, but the chat-interface approach remained vulnerable to robustness issues.Suggested remedies include knowledge-base conditioning, advanced decoding, and error handling that exposes logs and prompts reflection.Source: One sentence summaries · p. 58

C. whinchat: A Conversational ELN Interface

  • Whinchat interprets linked JSON from an experimental data system to summarize experiments, resolve shorthand, create tables or visualizations, and suggest follow-up experiments.In an example, it summarized NaCoO2 synthesis, electrode characterization, and battery cycling from the provided data.Source: C. whinchat: A Conversational ELN… · p. 59
  • Whinchat’s utility is task- and model-dependent, and its demonstrated context covered one project with up to 8 samples and 1700 JSON tokens.The authors propose fine-tuning on group knowledge and expanding context through larger models, embeddings, or API querying; retrieval-augmented generation is another proposed remedy.Source: C. whinchat: A Conversational ELN… · pp. 60–61, One sentence summaries · p. 61

D. BOLLaMa

  • BOLLaMa’s contribution is broadening access to chemical-reaction optimization, with further development proposed for safety recommendations and clearer explanations of the process.The project emphasizes that accessible tools should accommodate users with diverse technical backgrounds.Source: D. BOLLaMa · p. 62

One sentence summaries

  • BOLLaMa uses a natural-language chat interface to make Bayesian optimization accessible, initializing runs and translating reported observations into optimization-step calls.Its current implementation retrieves initialization and optimization-step functions on LLM demand, but correct functioning is not guaranteed for every prompt.Source: One sentence summaries · pp. 62–63, D. BOLLaMa · p. 62

A. InsightGraph

  • Guided LLM extraction can make structured literature graphs much faster to create than manual curation and let scientists without NLP expertise search and build on dispersed knowledge.The authors also suggest similar entity-and-relationship extraction could benefit other scientific fields.Source: A. InsightGraph · p. 64
  • InsightGraph prompted GPT-3.5-turbo to extract entities and relationships from 200 polymer-nanocomposite abstracts into a neo4j graph, using a constrained JSON schema.The prompt included an example schema, and the resulting JSON was stored using Cypher; schema constraints were needed because changing entity types required pipeline changes.Source: A. InsightGraph · p. 64, Figure 12 · p. 64
  • Constrained JSON reduced variable outputs, but InsightGraph still needs systematic precision-and-recall evaluation, and pairwise links may not capture materials’ complex relationships.The authors call for comparisons across models and prompting or fine-tuning approaches, as well as improved input schemas.Source: A. InsightGraph · pp. 64–65

One sentence summaries … 2. Solution

Across these projects, LLMs converted unstructured chemistry text and tables into structured data and generated scientific titles or abstracts. Their successes depended on task and input format, with schema compliance, special characters, and output quality remaining challenges.

One sentence summaries

  • GPT-3.5-turbo prompting produced meaningful JSON entities and relationships from hundreds of abstracts, but nondeterminism can make the outputs variable and fragile.The project called for more systematic benchmarking to understand performance and variability.Source: One sentence summaries · p. 65

B. Extracting Structured Data from Free-form Organic Synthesis Text

  • GPT-3 fine-tuned on 300 examples correctly extracted 93% of reaction-input components across 50 evaluation pairs, including properties and external compound references; Alpaca-LoRA did not produce valid ORD JSON.The model linked properties such as volume or mass to components, while extracting compound numbers referenced external entities.Source: B. Extracting Structured Data from… · p. 66, One sentence summaries · p. 67
  • Reaction-text extraction remains vulnerable to misplaced entities across reaction and workup descriptions, missing earlier context for compound references, and completion of names beyond the prompt.Addressing external references may require prompting with multiple paragraphs or an entire document, which adds irrelevant tokens.Source: B. Extracting Structured Data from… · p. 67

C. TableToJson: Extracting structured information from tables in scientific papers

  • For TableToJson, a table with a few catalyst names containing “-” yielded 94% accuracy with those values erroneously generated; power-number errors produced 80% initially, rising to 100% after a prompt explanation.Source: C. TableToJson: Extracting structured… · p. 70
  • 46% accuracy on a complex table rose to 60% by increasing maximum string-token length and to 86% with a power-number explanation, while hyphen handling varied from 86% to 100%.The table included long molecule names, hyphens, powers, subscripts, and superscripts; Unicode substitutions and other characters remained difficult to resolve systematically.Source: Figure 15 · p. 69, C. TableToJson: Extracting structured… · pp. 70–71
  • Both table-to-JSON approaches achieved 100% accuracy in most examples with regular strings or numbers, but direct prompting may depart from a schema and Jsonformer can struggle with special characters.Jsonformer ensures valid JSON syntax and schema compliance, while the OpenAI model can alter the requested schema; special-character failures were not systematically resolved.Source: C. TableToJson: Extracting structured… · p. 71, One sentence summaries · p. 72

1. Problem

  • Scientific text-generation tasks lack uniquely defined outputs and curated datasets, so the project used expert-written titles as benchmarks and ROUGE metrics for evaluation.Open-access chemistry and materials manuscripts were identified as potential data for generating abstracts from titles or titles from abstracts, with a human expert in the loop.Source: §1 · p. 73, §2 · p. 73

2. Solution

  • ChemNLP fine-tuned T5 for abstract summarization and GPT-2-medium for text generation, but generated abstracts contained grammatical and syntactic errors.The project also described a JARVIS-Leaderboard for community benchmarking; the authors suspected the errors reflected use of a small, outdated base model and called for more analysis.Source: §2 · pp. 73–74

One sentence summaries … Acronyms

The section describes LLM applications for scientific communication and education, from generating titles and abstracts to turning lecture recordings into summaries and learning resources. It also reports participant diversity and supplies an acronym glossary.

One sentence summaries

  • Fine-tuned T-5 and GPT-2 models showed initial promise for generating titles from abstracts and abstracts from titles, but expert evaluation is still needed.The authors call for more systematic analysis, including domain experts rating generated titles and abstracts.Source: One sentence summaries · p. 74

A. i-Digest

  • i-Digest turns a video or audio link into a transcript, technical keywords, a summary, questions, and links from mentioned chemicals to PubChem entries.The implementation uses Whisper for transcription and GPT-3.5-turbo for summaries and questions; Figure 19 depicts the interface and outputs.Source: A. i-Digest · p. 75, Figure 19 · p. 75
  • i-Digest aims to help learners and educators create or update course content, with resources designed for audiences from novice learners to professors and lifelong learners.Links to resources beyond the lecture are intended to encourage deeper understanding.Source: A. i-Digest · pp. 75–76

One sentence summaries

  • The system can transcribe content, generate meaningful questions, and extract chemical mentions, while lacking suitable benchmarks for systematic performance evaluation.Proposed extensions include conditioning on lecture notes or books and scoring answers to assess learning and direct students to relevant material.Source: One sentence summaries · p. 76

V. Meta analysis of the workshop contributions

  • Workshop coauthors had an approximately 30% female/male ratio and came from 22 institutions in 8 countries; around 20% of teams were international.Teams often combined expertise across institutions, and some included participants from outside academia.Source: §V · p. 77, Figure 20 · p. 77, Figure 21 · p. 77

Acronyms

  • The acronym glossary defines AI, API, BO, DFT, GPT, and LLM for readers across the paper’s projects.These entries expand to artificial intelligence, application programming interface, Bayesian optimization, density functional theory, generative pretrained transformer, and large language model.Source: Acronyms · p. 82

Related papers

Loading 2306.06283v4…