PaperlayerView exact revision on arXiv

Source-linked AI summary

14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon

Kevin Maik Jablonka, Qianxiang Ai, Alexander Al-Feghali, Shruti Badhwar, Joshua D. Bocarsly, Andres M Bran, Stefan Bringuier, L. Catherine Brinson, Kamal Choudhary, Defne Circi, Sam Cox, Wibe A. de Jong, Matthew L. Evans, Nicolas Gastellu, Jerome Genzling, María Victoria Gil, Ankur K. Gupta, Zhi Hong, Alishba Imran, Sabine Kruschwitz, Anne Labarre, Jakub Lála, Tao Liu, Steven Ma, Sauradeep Majumdar, Garrett W. Merz, Nicolas Moitessier, Elias Moubarak, Beatriz Mouriño, Brenden Pelkie, Michael Pieler, Mayk Caldas Ramos, Bojana Ranković, Samuel G. Rodriques, Jacob N. Sanders, Philippe Schwaller, Marcus Schwarting, Jiale Shi, Berend Smit, Ben E. Smith, Joren Van Herck, Christoph Völker, Logan Ward, Sean Warren, Benjamin Weiser, Sylvester Zhang, Xiaoqi Zhang, Ghezal Ahmad Zia, Aristana Scourtas, KJ Schmidt, Ian Foster, Andrew D. White, Ben Blaiszik

arXiv:2306.06283v4cond-mat.mtrl-scics.LGphysics.chem-ph

TL;DR

The paper investigates how LLMs can support chemistry and materials-science workflows, including flexible laboratory-data interfaces, molecular-energy prediction, visualization, structured datasets, and education. Hackathon prototypes demonstrated that diverse LLM applications could be developed rapidly, while model limitations include limited transparency into how the models were built.

  • Problem

    Traditional laboratory-data interfaces lack enough flexibility to capture rich, diverse, and interconnected experimental data.

  • Method

    The authors developed hackathon prototypes using LLMs for molecular-energy prediction, natural-language visualization interfaces, and conversational laboratory-data management.

  • Results

    Working LLM prototypes spanning scientific prediction, visualization, data management, structured datasets, and education were generated in a short time.

  • Takeaways & Limitations

    LLMs can support applications including flexible conversational interfaces, custom scientific visualizations, structured materials-specific datasets, and educational tools.

  • Takeaways & Limitations

    The models have limited transparency because users cannot examine how they were built.

Abstract

from arXiv · show

Large-language models (LLMs) such as GPT-4 caught the interest of many scientists. Recent studies suggested that these models could be useful in chemistry and materials science. To explore these possibilities, we organized a hackathon. This article chronicles the projects built as part of this hackathon. Participants employed LLMs for various applications, including predicting properties of molecules and materials, designing novel interfaces for tools, extracting knowledge from unstructured data, and developing new educational applications. The diverse topics and the fact that working prototypes could be generated in less than two days highlight that LLMs will profoundly impact the future of our fields. The rich collection of ideas and projects also indicates that the applications of LLMs are not limited to materials science and chemistry but offer potential benefits to a wide range of scientific disciplines.

6Laboratory of Artificial Chemical Intelligence (LIAC), Institut des Sciences et Ing´enierie Chimiques,

The hackathon explored LLM applications across chemistry and materials science, producing working prototypes for prediction, interfaces, knowledge extraction, and education. Its breadth and rapid development suggest broad future relevance, although the prototypes remain early-stage.

  • LLMs can use natural-language inputs to extract structured information, connect existing tools, and support flexible interfaces.
  • The projects addressed applications from wet-lab and computational chemistry workflows to software interfaces and classrooms.
  • The authors conclude that LLM-based projects that once required many months could be created in hours, while remaining unpolished prototypes.
  • Working prototypes were developed during a 1.5-day collaborative effort spanning eight countries and 22 institutions.
  • The hackathon organized projects into predictive modeling, novel interfaces and automation, knowledge extraction, and education.
  • Knowledge-extraction projects show that LLMs can make information more accessible by extracting structured information from unstructured text.

A. Predictive modeling

This section extends earlier chemistry applications of LLMs by examining predictive modeling with language-interfaced data and established or novel modeling concepts.

  • The projects build on earlier demonstrations that LLMs can predict chemical properties and guide experiments through in-context learning.
  • The LIFT framework can include in-context learning settings where structured data are converted into text prompts for an LLM.

a. Molecular Energy Predictions

The predictive case studies apply LLMs to molecular atomization energies and concrete strength. They show competitive prediction and the incorporation of qualitative domain knowledge, while identifying accuracy and evaluation limits.

  • Molecular Energy Predictions: LIFT using SMILES and SELFIES achieved R2 > 0.95 on a holdout test set, but remained inferior to dedicated models using 3D information.
  • Molecular Energy Predictions: A Δ-ML scheme enabled LIFT models using string representations to reach chemical accuracy for molecular atomization-energy prediction.
  • Molecular Energy Predictions: The LIFT approach could also be implemented with GPT-2 and LoRA on consumer hardware, improving accessibility without relying on the OpenAI API.
  • Text2Concrete: Text2Concrete used few-shot prompting to predict concrete compressive strength from formulation data, with accuracy comparable to GPR but inferior to RF.
  • Text2Concrete: Adding the natural-language rule that high water/cement ratio reduces strength reduced outliers and produced R2 values of 0.67 for LLMs and 0.72 for RF.
  • Text2Concrete: The Text2Concrete study did not exhaustively analyze how changing fuzzy context affects LLM performance, leaving this as future research.

c. Molecule Discovery by Context

This section uses contextual scientific text and LLM embeddings to identify candidate hydrogen-carrier molecules, while also describing prompt paraphrasing to augment LIFT training data.

  • Molecule Discovery by Context: Contextual embeddings represent words according to their sentence context, unlike static Word2Vec embeddings that remain unchanged after training.
  • Molecule Discovery by Context: ScholarBERT sentence embeddings were averaged into molecular fingerprints that ranked candidate molecules by similarity to known hydrogen-carrier molecules.
  • Molecule Discovery by Context: Visual inspection indicated that selected molecules resembled known hydrogen-carrier molecules.
  • Prompt augmentation: LLMs can paraphrase prompt templates to create high-quality training prompts, augment datasets, and reduce overfitting to one template.
  • Prompt augmentation: Prompt augmentation may help retain general language abilities after fine-tuning, but its effects remain a stated research question.

e. Genetic algorithm using an LLM

The McGill team explored using LLMs to perform genetic-algorithm operations on molecules represented by SMILES. Initial experiments found that GPT-3.5 could fragment molecules and generate chemically reasonable recombinations, while broader applicability remains constrained by SMILES validity and context length.

  • Initial experiments: GPT-3.5 fragmented SMILES molecules at rotatable bonds with a 70% success rate without fine-tuning.The result indicates that GPT-3.5 captures aspects of the relationship between SMILES strings and chemical structures.
  • LLM genetic operations: The LLM fragmented and recombined two molecules, producing combined molecules containing fragments from both parents.This operation corresponds to reproduction or crossover in a genetic algorithm.
  • Evaluation: Organic chemists judged LLM-generated molecules chemically reasonable in 32/32 cases, compared with 21/32 for random recombination.The comparison concerns expert judgments of chemical reasonableness in the tested cases.
  • Optimization: A preliminary visual inspection suggested that prompting an LLM with parent molecules and performance scores could yield chemically reasonable modifications.The task asked the LLM to propose molecules improving Tanimoto similarity to vitamin C.
  • Implications: The results indicate that suitably conditioned LLMs might reproduce known structures and generate new structures that make chemical sense.The authors frame this as an indication rather than a systematic demonstration of improvement.
  • Limitations: Most LLMs still struggle to output valid SMILES without explicit fine-tuning, and context limits restrict the number of parent molecules supplied as examples.The authors suggest chemistry-specific foundation models as one possible mitigation.

B. Automation and novel interfaces

The projects demonstrate LLMs acting as natural-language interfaces and agents for chemistry and materials-science software. They translate questions into workflows, connect external tools, and make specialized tools more accessible, although robustness and user training remain challenges.

  • Tool-using agents: LLM agents can use external tools and step-by-step reasoning to answer questions based on observations from robust tools.This approach is described as converting hyperconfident models that hallucinate into systems grounded in tool queries.
  • Accessibility: LLM interfaces can lower the barrier to protein-engineering tools and connect programs through natural language instead of tool-specific APIs or glue code.The discussion includes extensions to VMD, voice control, and possible PyMOL interfaces.
  • MAPI-LLM: MAPI-LLM converts text prompts into Materials Project API queries for stability, classification, and regression questions.It can create multi-step workflows, such as checking metallicity before calculating a band gap.
  • MAPI-LLM: MAPI-LLM uses in-context learning from similar Materials Project entries when a requested property is unavailable, bridging data gaps.The resulting context is used by an LLM to infer properties for the unknown material.
  • Visualization interfaces: Few-shot prompting enabled a prototype that retrieves protein structures and generates custom 3dmol.js visualizations from natural-language commands.The LLM translates varied wording, including spelling variants and color descriptions, into interpretable visualization instructions.
  • Limitations: Robustness issues include prompt fragments leaking into outputs, while developing these applications requires scientists to invest substantial time in learning the underlying systems.The authors suggest retries, prompt engineering, and documentation-based knowledge bases as improvement routes.

c. ELN interface: whinchat

The whinchat project embeds an LLM assistant in a materials-chemistry data-management system to interpret and act on structured and unstructured experimental records. The broader examples also show LLM interfaces supporting Bayesian optimization and workflow automation.

  • Motivation: Traditional interfaces struggle to capture the flexibility and interconnectedness of experimental data, whereas LLMs can interpret structured and unstructured records.This motivates conversational interfaces for electronic laboratory and materials data.
  • Whinchat: Whinchat attaches a chat assistant to samples in datalab and accesses structured and free-text data through its JSON API.The system is embedded in an open-source materials chemistry data-management platform.
  • Whinchat: Whinchat contextualizes experiments, resolves acronyms, summarizes nested entries, and reformats data into flowcharts or Markdown tables.These functions use linked responses and can render experimental information in alternative formats.
  • Whinchat: Whinchat can suggest related materials, synthesis protocols, characterization techniques, and NMR-active nuclei to probe.The NMR example is reported in the supporting-information examples.
  • Limitations: The current implementation is limited by context-window size, restricting each conversation to a relatively small number of samples.Potential responses include embeddings for sample selection or an agent that queries the ELN API between extraction and summarization calls.
  • Bayesian optimization: BoLLama provides a chat interface that initializes Bayesian optimization, incorporates user feedback, and recommends subsequent experiments.Persisted experimental results could be used to bootstrap later optimization runs.

C. Knowledge Extraction

The projects use LLMs to search, summarize, structure, and visualize chemical literature and experimental procedures. These capabilities can support knowledge graphs and materials-specific datasets, while extraction quality depends on prompting or fine-tuning approaches.

  • Literature search: LLMs can rapidly mine chemical literature by retrieving, summarizing, and contextualizing relevant fragments from large document collections.Paper-QA is given as an example that answers questions with citations from libraries of hundreds of documents.
  • Structured extraction: Iterative fine-tuning uses domain-expert corrections to improve JSON extraction from scientific papers.Initial zero-shot completions are corrected and then used to fine-tune the LLM.
  • Structured extraction: Careful prompting can produce powerful prototypes for converting unstructured chemical information into structured data.The paper presents InsightGraph as an example of prompt-based knowledge-graph construction.
  • Knowledge graphs: GPT-3.5-turbo generated knowledge-graph representations of materials and pairwise relationships from polymer-nanocomposite text using an example JSON schema.The resulting representations can be visualized with graph tools.
  • Downstream uses: Knowledge graphs could help compare material types and uses across articles and launch literature reviews.The text also identifies structured, materials-specific datasets as a downstream application for question answering and machine learning.
  • Organic synthesis parsing: Fine-tuning on 300 prompt-completion pairs enabled extraction of 93% of reaction components from free-text procedures.The Organic Synthesis Parser targets structured JSON conversion for reaction data.
  • Organic synthesis parsing: Structured reaction data can expand training data for predicting reaction conditions and yields.The Open Reaction Database provides curated reactions with procedures also available as plain text.

c. TableToJson: Structured information from tables in scientific papers

TableToJson uses LLMs to extract scientific-table information into schema-defined JSON, supporting structured data workflows while revealing schema-following and notation-related limitations.

  • OpenAI model: OpenAI text-davinci-003 extracted table-cell information effectively and generated JSON objects, but did not always follow the provided schema.The model produced valid JSON objects, yet schema compliance was not guaranteed.
  • Jsonformer: Jsonformer reads schema keys and generates only value tokens, guaranteeing syntactically valid JSON corresponding to the desired schema.This decoding strategy addresses the inability of unconstrained LLM generation to guarantee valid JSON syntax.
  • Implications: LLMs can extract scientific information into fixed-schema structured data that can be stored in databases and support topic-specific database creation.The authors position this capability as useful for automated analysis, extraction, and integration into computational workflows.

D. Education

The hackathon demonstrated educational and scientific applications for LLMs, including digital tutoring, tool development, and work with unstructured data. The prototypes also highlighted unresolved concerns about robustness, interpretability, evaluation, reproducibility, and safe deployment.

  • I-Digest: I-Digest uses lecture recordings and course material to provide students with a digital tutor.Whisper transcribes videos, after which an LLM generates questions and answers and can identify chemicals for links to further information.
  • I-Digest: The system can generate practically infinite questions, use student feedback for improvement, and potentially recommend exercises or relevant learning materials.Questions may be presented before or after videos to support skipping known material or directing students to relevant timestamps and additional content.
  • Applications: The prototypes show that LLMs can support novel chemistry and materials-science workflows by incorporating fuzzy design rules and operating on unstructured data.The projects also reused established concepts such as Δ-ML while applying LLMs to new problem settings.
  • Tool development: LLMs helped convert fuzzy abstractions into code, producing tools with shorter, more understandable code and fewer dependencies.The authors suggest that natural-language descriptions of existing solutions might support interoperability without requiring additional formats or standards.
  • Limitations and future needs: LLM applications remain constrained by limited interpretability, lack of robustness, fragile zero- and few-shot performance, and insufficient evaluation and reproducibility frameworks.The authors call for benchmarks beyond tabular datasets, automated tests, and versioning of training data, models, tools, and prompting setups.
  • Applications: The projects did not rely on LLMs’ chemistry knowledge; instead, they used general reasoning with chemistry information supplied through context or fine-tuning.This pattern distinguishes the demonstrated applications from approaches that depend on the models’ internal chemistry understanding.

Supporting Information for: 14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model

The supporting information lists the hackathon's contributors and their affiliations across universities, research institutions, companies, and independent researchers.

  • Affiliations: The author list includes contributors affiliated with EPFL and its Laboratory of Molecular Simulation.Additional EPFL-linked affiliations include the Laboratory of Artificial Chemical Intelligence and NCCR Catalysis.
  • Affiliations: Contributors also represent MIT, McGill University, the University of Cambridge, and several other universities.Listed institutions include Duke University, the University of Rochester, UCLouvain, and the University of Lausanne.
  • Affiliations: The contributors additionally include researchers from companies, national laboratories, and independent research organizations.Affiliations include Reincarnate Inc., the National Institute of Standards and Technology, Lawrence Berkeley National Laboratory, and independent researchers.

I. Predictive Modeling

The paper presents LLM-based predictive modeling for molecular energies, concrete strength, and molecule discovery, using string representations, contextual knowledge, and similarity-based screening. These projects show that LLMs can support accurate or efficient prediction, although performance depends on the modeling approach and available molecular information.

  • Molecular energy prediction: LLMs were fine-tuned on SMILES or SELFIES strings to predict molecular atomization energies at the G4(MP2) level.The workflow used molecular string representations as prompts and evaluated predictions on held-out molecules.
  • Limitations: LLM prediction performance can be fragile in zero- or few-shot settings, and molecular string inputs do not fully represent 3D environments.The authors propose testing other models and richer molecular-input representations.
  • Molecular energy prediction: A ∆-ML approach brought atomization-energy errors close to dedicated baselines, whereas direct fine-tuning produced errors one order of magnitude higher.The paper identifies this as the first demonstration of ∆-ML in the LIFT framework for chemistry.
  • Contextual materials prediction: With qualitative concrete-design knowledge, in-context learning reduced prediction outliers and surpassed RF, reaching R-squared = 0.71.Without this context, ICL achieved R-squared values of 0.5 and 0.54 in the reported small-data comparisons, while RF reached 0.67.
  • Molecule discovery: The LLM screening approach evaluated about 100 molecules per second on a V100 GPU versus roughly 30 seconds per molecule for traditional quantum chemistry, a 3000 times speedup.The reported success metrics were hydrogen capacity and energy penalty for adding and removing H2.
  • Molecule discovery: ScholarBERT-based similarity screening recommended hydrogen-carrier molecules better than random and identified structural or literature similarities such as 5-member rings.Candidates were ranked by cosine similarity between ScholarBERT representations of known and candidate molecules.

3. Impact

The projects explored LLMs for molecular optimization, materials-question answering, and tool-assisted scientific workflows. Early results suggest useful capabilities alongside robustness and evaluation gaps.

  • Molecular optimization: GPT-3.5 frequently fragmented druglike molecules into valid SMILES strings, but produced non-original fragments for 2/10 molecules and failed valid generation for 1/10 after ten tries.The failures were consistent across multiple runs and involved difficulty understanding some SMILES strings.
  • Molecular optimization: An LLM-as-a-GA method prompted the model with scored molecules to generate new candidates for molecular property optimization.The vitamin C task used few-shot examples and iterative generation of SMILES strings.
  • Molecular optimization: LLM-guided molecular modifications appeared more chemically sound than quasi-random genetic-algorithm operations, suggesting potential for more efficient genetic operations.The evidence was based on visual inspection, and systematic robustness comparisons remain necessary.
  • Materials tools: MAPI-LLM combined an LLM agent, Materials Project API, in-context learning, and reaction-network tools to answer materials questions and suggest synthesis or decomposition reactions.Tool coupling allowed questions to be addressed beyond the capabilities of individual tools or the LLM alone.
  • Materials tools: LLM agents remained uneven across prompts, while incorporating tool descriptions limited the number of tools that could be coupled.The authors call for systematic investigation to improve robustness.

One sentence summaries

The projects used conversational LLM interfaces to make visualization, laboratory data, Bayesian optimization, and knowledge-graph tools more accessible. They also exposed limitations involving robustness, context size, and task-dependent usefulness.

  • Tool interfaces: A chat interface prompted an LLM to generate 3dmol.js commands, often performing requested molecular-visualization actions without consulting documentation.The approach targeted non-experts who lacked the coding skills needed to customize visualizations.
  • Laboratory data: Whinchat let users query experimental data flexibly by producing summaries, tables, CSV files, mermaid.js visualizations, and suggested future experiments.It could also contextualize linked data and identify related materials, additives, characterization techniques, and synthesis protocols.
  • Laboratory data: An LLM assistant summarized experimental data and identified NMR-active nuclei, while NMR’s sample requirements and incomplete nuclear sensitivity necessitate complementary techniques.The paper notes that scientific usefulness is highly task- and model-dependent.
  • Optimization interfaces: BOLLaMa initialized Bayesian-optimization runs through chat and converted natural-language observations into calls to the optimizer’s stepping function.The project addressed programming and parameter-access barriers to chemical optimization tools.
  • Knowledge extraction: LLM-based assistants could create structured knowledge graphs and lower barriers to searching and building on scientific knowledge.The paper contrasts rapid graph generation with the years required for manual curation, while noting hallucination risk without structured guidance.

One sentence summaries

The projects extracted structured chemical information from abstracts, synthesis procedures, and scientific tables using prompted or fine-tuned language models. Performance was promising but sensitive to model behavior, formatting, and input complexity.

  • Text extraction: GPT-3.5-turbo generated meaningful JSON structures containing extracted entities and relationships for hundreds of abstracts.The authors identify nondeterministic behavior and fragile outputs as reasons for further benchmarking.
  • Organic synthesis: Fine-tuned OpenAI Davinci correctly extracted 93% of reaction-input components from 50 evaluation procedures using only 300 prompt-completion pairs.The model also associated properties such as reaction volumes or masses with extracted components.
  • Organic synthesis: The synthesis parser remained vulnerable to ambiguous reaction-versus-workup boundaries, external references, and unsupported completion of chemical names.These issues could place entities in incorrect fields or add information not explicitly present in the prompt.
  • Table extraction: The table-extraction approaches used scientific-table inputs, JSON schemas, and validation procedures to convert tabular content into structured data.The inputs included HTML table code, and outputs were parsed and validated with Python tools.
  • Table extraction: Table-to-JSON extraction achieved 100% accuracy in most regular-string or number examples, but complex tables reduced JSON-generation accuracy to 46%.Errors involved hyphens, powers, long molecule names, subscripts, and superscripts.

One sentence summaries

The hackathon also produced educational applications and a meta-analysis of participation. These projects generated lecture summaries, questions, chemical links, and cross-institutional collaboration, but evaluation remained limited.

  • Workshop meta-analysis: The projects addressed text summarization and generation, educational task creation, and multiple LLM-enabled scientific interfaces and data workflows.The paper reports these applications across chemistry and materials-science contexts.
  • Educational applications: A video and audio tool transcribed lectures, generated technical keywords and summaries, produced questions, and extracted chemical mentions from content.The implementation used Whisper for transcription and GPT-3.5-turbo for summaries, questions, and chemical extraction.
  • Educational applications: The educational tool was intended to support course creation, content updates, and learning across audiences ranging from novices to professors.It also linked users to databases, journals, and books for deeper study.
  • Educational applications: Systematic evaluation of the educational tool was difficult because suitable benchmarks were unavailable.Suggested extensions included conditioning on additional materials and automatically scoring answers.
  • Workshop meta-analysis: The workshop included participants from 22 institutions across 8 countries, with most teams combining expertise from different institutions.Around 20% of teams were international, and several included participants beyond academia.
Loading 2306.06283v4…