LLMatDesign: Autonomous Materials Discovery with Large Language Models

Shuyi Jia; Chao Zhang; Victor Fung

Source-linked AI summary

LLMatDesign: Autonomous Materials Discovery with Large Language Models

Shuyi Jia, Chao Zhang, Victor Fung

arXiv:2406.13163v1Jun 19, 2024cond-mat.mtrl-scics.AIcs.CL

TL;DR

Materials discovery spans an enormous chemical space, and data-driven approaches often rely on extensive training data that may be unavailable or costly to produce. LLMatDesign uses LLM agents to modify materials, evaluate outcomes with computational tools, and reflect on prior decisions. The authors report that it consistently outperforms baselines in materials-design experiments, while current examples focus on composition modifications.

Problem
Materials discovery is challenging across vast composition and structure spaces, and many data-driven methods rely on extensive training datasets that may be unavailable or constrained by experiment and simulation budgets.
Source: §1 · p. 1
Method
LLMatDesign uses LLMs to interpret natural-language design requirements, propose material modifications with hypotheses, evaluate outcomes using computational tools, and incorporate self-reflection into an iterative design process.
Source: §1 · p. 2, §2.1 · p. 3
Results
LLMatDesign consistently outperforms baselines in materials-design experiments, supporting its effectiveness in developing new materials.
Source: §2.6 · p. 10, §3 · p. 11
Takeaways & Limitations
The results highlight the potential of LLM-guided materials discovery in computational settings and, prospectively, toward robotic laboratories.
Source: §1 · p. 2
Current examples modify material composition; extending the framework to structural modifications is left for future work.
Source: §2.6 · p. 11

Abstract

from arXiv · show

Discovering new materials can have significant scientific and technological implications but remains a challenging problem today due to the enormity of the chemical space. Recent advances in machine learning have enabled data-driven methods to rapidly screen or generate promising materials, but these methods still depend heavily on very large quantities of training data and often lack the flexibility and chemical understanding often desired in materials discovery. We introduce LLMatDesign, a novel language-based framework for interpretable materials design powered by large language models (LLMs). LLMatDesign utilizes LLM agents to translate human instructions, apply modifications to materials, and evaluate outcomes using provided tools. By incorporating self-reflection on its previous decisions, LLMatDesign adapts rapidly to new tasks and conditions in a zero-shot manner. A systematic evaluation of LLMatDesign on several materials design tasks, in silico, validates LLMatDesign's effectiveness in developing new materials with user-defined target properties in the small data regime. Our framework demonstrates the remarkable potential of autonomous LLM-guided materials discovery in the computational setting and towards self-driving laboratories in the future.

1 Introduction

Materials discovery is constrained by vast chemical and structural spaces, while many data-driven methods depend on extensive training data that may be unavailable or costly to obtain. LLMatDesign proposes a flexible, LLM-driven design loop that uses computational evaluation and self-reflection to adapt to user-defined tasks.

1 Introduction

Materials discovery spans vast composition and structure spaces, while traditional screening can be time-consuming and resource-intensive and data-driven methods rely heavily on extensive training datasets.Such data may be unavailable or limited when experiment and high-fidelity simulation budgets are constrained.Source: §1 · p. 1
LLMatDesign’s workflow starts from user-provided composition and target property, recommends modifications, and uses machine-learning tools for structure relaxation and property prediction in a reflective loop.The overview depicts past modifications feeding back into subsequent decisions.Source: Figure 1 · p. 2, §1 · p. 2
LLMatDesign uses LLMs to interpret human instructions, propose material modifications with hypotheses, evaluate outcomes using computational tools, and incorporate self-reflection into iterative decisions.Users can adapt the framework to different tasks and constraints by modifying prompts.Source: §1 · p. 2
Unlike many data-driven generative methods, LLMatDesign is presented as eliminating the need for large training datasets derived from ab initio calculations.Its natural-language interface is intended to support adaptation across materials, tasks, and target properties through prompt changes.Source: §1 · p. 2

2 Results

LLMatDesign iteratively modifies materials using LLM-generated suggestions, tool-based evaluation, and feedback from prior decisions. Across the reported tasks, it reaches target properties, while reflection, prompt design, and natural-language constraints shape its performance and scope.

2.1 LLMatDesign Framework

LLMatDesign selects addition, removal, substitution, or exchange edits, evaluates modified structures with ML force fields and property predictors, then feeds reflection and modification history into the next decision.It can retrieve a starting structure from Materials Project when only the composition is specified; optional DFT calculations can evaluate the final material.Source: §2.1 · pp. 3–4

2.2 Evaluation

GPT-4o with history reached 1.4 eV in 10.8 modifications versus 27.4 for random; GPT-4o variants averaged −1.97 and −1.99 eV/atom formation energy.GPT-4o with history also achieved a minimum formation energy of −2.72 eV/atom.Source: Table 1 · p. 5, Table 2 · p. 5, §2.2 · pp. 5–6
LLM modification choices concentrate on selected elements, unlike random sampling; both LLMs usually suggest chemically viable elements, while Gemini-1.0-pro occasionally suggests noble gases.GPT-4o-generated structures averaged −2.32 eV/atom in DFT validation versus −1.51 eV/atom for random, with completion rates of 73.3% and 40.0%, respectively.Source: §2.2 · pp. 6–7, Figure 5 · p. 7
In an example targeting 1.40 eV, LLMatDesign successively substituted Se for S, Si for Ge, and Zn for Cu, using reflections to guide later steps.The first substitution reduced the band gap contrary to the goal; the third brought it within the acceptable threshold.Source: §2.2 · p. 7, Figure 6 · p. 8

2.3 Self-reflection

With GPT-4o, history plus self-reflection required 10.8 modifications to reach the band-gap target, versus 23.4 when reflection was omitted.The history message without reflection retained modification and hypothesis pairs; the authors report that reflection improves efficiency.Source: §2.3 · p. 9, Table 3 · p. 9

2.4 Prompting

GPT-4o Refined reached the band-gap target in an average of 8.69 modifications, outperforming both the original history prompt and the Persona template.GPT-4o itself created the refined prompt by rephrasing and reformatting the original and adding a step-by-step instruction.Source: §2.4 · p. 9, Table 4 · p. 10

2.5 Constrained Materials Design

For SrTiO3, LLMatDesign complied with “do not use Ba or Ca” and “do not modify Sr” in 100% of modifications, and with the four-element limit in 99.02%.Four of 509 modifications violated the four-distinct-elements constraint.Source: §2.5 · p. 10, Table 5 · p. 10

2.6 Further Discussion

The reported designs use a limited set of composition modifications in zero-shot settings; future work would add structural information and edits to atomic positions and lattice.The authors also describe chemical fine-tuning as a possible extension for generating more insightful hypotheses and explanations.Source: §2.6 · pp. 10–11

3 Conclusion

LLMatDesign uses natural-language design requirements and computational tools to automate materials design, adapting to new tasks through prompt changes and learning from prior iterations. The experiments report consistent outperformance of the baseline, while the authors point to future integration with autonomous laboratories.

LLMatDesign works directly with user-defined design requirements and constraints expressed in natural language.Source: §3 · p. 11
It integrates computational tools for structure relaxation and property evaluation and learns from previous iterations.Source: §3 · p. 11
Modifying the prompt lets LLMatDesign adapt to different tasks, target properties, and design constraints.Source: §3 · p. 11
In the reported experiments, LLMatDesign consistently outperforms the baseline.Source: §3 · p. 11
The authors highlight its potential for integration into autonomous laboratories in the future.Source: §3 · p. 11

4 Methods

The methods use LLMs alongside machine-learning models for force-field relaxation and property prediction. Materials are modified through four operation types, then relaxed and evaluated computationally.

4.1 Large Language Models

GPT-4o and Gemini-1.0-pro are the demonstrated LLMs; the paper describes LLMs as transformer-based models trained on text.GPT-4o is reported with a 128K context length and an October 2023 knowledge cutoff; Gemini-1.0-pro has a 32K context length.Source: §4.1 · p. 11

4.2 Machine Learning Force Field

The TorchMD-Net force field is trained on 187,687 Materials Project crystal structures with energies, forces, and stresses.Training uses MatDeepLearn for 400 epochs on one Nvidia A100 80GB GPU.Source: §4.2 · p. 11

4.3 Machine Learning Property Predictor

TorchMD-Net property predictors target band gap and formation energy per atom using MatBench datasets of 106,113 and 132,752 structures, respectively.Each model is trained for 200 epochs on one Nvidia A100 80GB GPU.Source: §4.3 · pp. 11–12

4.4 Modification of Material

LLMatDesign applies exchange, substitute, remove, or add operations directly to an ase.Atoms object, then relaxes the modified structure with an MLFF.An exchange operation replaces each specified element with the other.Source: §4.4 · p. 12

4.5 Modification of Material

DFT calculations use VASP with settings specified by Pymatgen's MPRelaxSet, as used in Materials Project.Source: §4.5 · p. 12

A Prompt Templates for LLMatDesign

The appendix shows prompt templates for Gemini-1.0-pro and GPT-4o, plus a template using a materials-design-expert persona. The templates distinguish task-specific placeholders from optional ones.

Gemini-1.0-pro uses a slightly different prompt template because it inconsistently generates standardized output.Source: A Prompt Templates for LLMatDesign · p. 15
The Gemini-1.0-pro template uses red placeholders for task-specific text and blue placeholders for optional text.Source: Figure A.1 · p. 15
The GPT-4o refined template likewise marks task-specific placeholders in red and optional placeholders in blue.Source: Figure A.2 · p. 16
A separate template adds a materials-design-expert persona while retaining the red task-specific and blue optional placeholder scheme.Source: Figure A.3 · p. 16

B Convergence Plots

The appendix plots average band gaps and formation energies over 50 modifications for ten starting materials, with separate figures for GPT-4o and Gemini-1.0-pro. The band-gap figures mark a 1.4 eV target and show average modification counts on the x-axis.

The GPT-4o band-gap plot averages results over 50 modifications for ten starting materials, marks the 1.4 eV target, and uses colored x-axis dots for average modifications to target.Source: Figure B.1 · p. 17
The Gemini-1.0-pro band-gap plot uses the same 50-modification, ten-material scope, 1.4 eV target line, and x-axis modification-count dots.Source: Figure B.2 · p. 17
The GPT-4o formation-energy plot averages over 50 modifications for ten starting materials and targets the lowest possible formation energy per atom.Source: Figure B.3 · p. 18
The Gemini-1.0-pro formation-energy plot uses the same 50-modification, ten-material scope and lowest-possible-energy objective.Source: Figure B.4 · p. 18

C Heatmaps

The heatmaps show element frequencies in the band gap and formation energy tasks for Gemini-1.0-pro and GPT-4o without history, using log-scaled colors to represent occurrence across modified materials.

The heatmaps cover band gap and formation energy tasks for Gemini-1.0-pro and GPT-4o without history.Source: Figure C.1 · p. 19
Element frequencies count occurrences in intermediate and final modified materials across all runs and starting materials.Source: Figure C.1 · p. 19
Darker colors indicate higher element frequencies, while lighter colors indicate lower frequencies or absence; the visualization uses log-scaling.Source: Figure C.1 · p. 19

D Visualization of Selected Structures

The figures visualize structures produced for band gap and formation energy tasks using GPT-4o with history, showing results from the first run across 10 starting materials.

For the band gap task, the figure shows final structures from the first run of all 10 starting materials and their corresponding band gaps.Source: Figure D.1 · p. 20
Both figures identify starting-material formulae in red and use GPT-4o with history as the LLM engine.Source: Figure D.1 · p. 20, Figure D.2 · p. 21
For the formation energy task, the figure shows first-run structures with the minimum formation energy per atom for each of 10 starting materials.Source: Figure D.2 · p. 21

E DFT Calculations

The table presents DFT results for lowest-energy structures from the formation energy task, averaged across starting materials and runs.

The table reports DFT results for lowest-energy structures obtained from the formation energy task.Source: Table E.1 · p. 22
The reported results are averaged across all starting materials and runs.Source: Table E.1 · p. 22
The table’s scope is the lowest-energy structures from the formation energy task, rather than all structures produced.Source: Table E.1 · p. 22