PaperlayerView exact revision on arXiv

Source-linked AI summary

LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

Yuan Chiang, Elvis Hsieh, Chia-Hong Chou, Janosh Riebesell

arXiv:2401.17244v3cs.CLcond-mat.mtrl-scics.AI

TL;DR

Scientific applications of LLMs face hallucinations and limited retained or up-to-date knowledge. LLaMP uses hierarchical ReAct agents to retrieve and work with materials information; its benchmarks report the highest SCoR and lowest errors across evaluated workflows.

  • Problem

    LLMs can hallucinate with high confidence and often lack up-to-date factual knowledge beyond their training data, challenging reliable scientific use.

    Source: §1 · p. 1

  • Method

    LLaMP is a multimodal RAG framework using hierarchical ReAct agents to interact with materials data and tools, without fine-tuning.

    Source: §1 · p. 2

  • Results

    LLaMP consistently achieves the highest SCoR and lowest material-property errors across the evaluated workflows and models.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    The framework lowers entry barriers to computational materials and chemistry and offers users a way to navigate and manipulate complex materials informatics.

    Source: §1 · p. 2, §6 · p. 9

    Correctness depends on backbone function-calling and reasoning, theoretical prediction quality, and Materials Project coverage; GGA bandgaps underestimate experimental values by 40–50%.

    Source: §5.2 · p. 6, §6 · p. 9

Abstract

from arXiv · show

Reducing hallucination of Large Language Models (LLMs) is imperative for use in the sciences, where reliability and reproducibility are crucial. However, LLMs inherently lack long-term memory, making it a nontrivial, ad hoc, and inevitably biased task to fine-tune them on domain-specific literature and data. Here we introduce LLaMP, a multimodal retrieval-augmented generation (RAG) framework of hierarchical reasoning-and-acting (ReAct) agents that can dynamically and recursively interact with computational and experimental data on Materials Project (MP) and run atomistic simulations via high-throughput workflow interface. Without fine-tuning, LLaMP demonstrates strong tool usage ability to comprehend and integrate various modalities of materials science concepts, fetch relevant data stores on the fly, process higher-order data (such as crystal structure and elastic tensor), and streamline complex tasks in computational materials and chemistry. We propose a simple metric combining uncertainty and confidence estimates to evaluate the self-consistency of responses by LLaMP and vanilla LLMs. Our benchmark shows that LLaMP effectively mitigates the intrinsic bias in LLMs, counteracting the errors on bulk moduli, electronic bandgaps, and formation energies that seem to derive from mixed data sources. We also demonstrate LLaMP's capability to edit crystal structures and run annealing molecular dynamics simulations using pre-trained machine-learning force fields. The framework offers an intuitive and nearly hallucination-free approach to exploring and scaling materials informatics, and establishes a pathway for knowledge distillation and fine-tuning other language models. Code and live demo are available at https://github.com/chiang-yuan/llamp

1 Introduction

Scientific use of LLMs is challenged by hallucinations and limited access to retained, up-to-date factual knowledge. LLaMP addresses this with external data and hierarchical ReAct agents for multimodal materials tasks.

1 Introduction

  • LLMs can provide outright false information with high confidence, a particular concern in knowledge-intensive scientific tasks requiring accurate guidance.The introduction also notes that LLMs often lack up-to-date factual knowledge outside their training data.Source: §1 · p. 1
  • Fine-tuning and tailored prompts have reproducibility, scalability, memory-retention, or generalizability constraints, motivating external data sources for factual responses.Fine-tuned models may forget knowledge during continued training and cannot know data added after pretraining.Source: §1 · pp. 1–2
  • LLaMP uses multimodal RAG and hierarchical ReAct agents to access materials and literature sources, retrieve higher-order data, and support complex inferences and operations without fine-tuning.The stated contributions include evaluating key material properties, synthesis and structure applications, and high-throughput atomistic workflows with pre-trained force fields.Source: §1 · p. 2

2 Background

The Materials Project provides calculated properties and analysis tools for inorganic materials and molecules. Scientific NLP also includes tools that turn crystal structures into textual descriptions.

2 Background

  • The Materials Project uses high-throughput DFT calculations and provides calculated material properties through API endpoints.Its workflow includes standardized structure relaxation and post-processing for properties such as bandgaps and elastic tensors.Source: §2 · p. 2
  • The Materials Project draws most atomic structures from the Inorganic Crystal Structure Database and aims to reduce materials-discovery time and cost through computational screening.The stated goal is to focus experiments on promising candidates identified by screening.Source: §2 · p. 2
  • Robocrystallographer converts crystal structures into descriptive JSON representations, and the Materials Project makes descriptions for 130K compounds available through its website and API.The representations include coordination statistics, connectivity motifs, geometric features, and dimensionality.Source: §2 · p. 2

3 Related Work

Prior domain-science approaches include prompt-based extraction, fine-tuned models, and tool-using agents. The paper identifies flat tool planning as a limitation and motivates hierarchical ReAct agents.

3 Related Work

  • Prompt-based methods have supported targeted extraction and chemistry tasks, but fine-grained prompts can reduce reproducibility and generalizability and remain prone to hallucination.Examples include extracting material properties and searching selected papers for synthesis information.Source: §3 · pp. 2–3
  • Fine-tuned models have been applied to molecular representations, materials chemistry, and alloy or molecule tasks, but may lack awareness of post-training results and forget during continual training.The cited examples include ChemGPT, GPT-3 fine-tuned on online corpora, and training on about 500 prompt-completion pairs.Source: §3 · p. 3
  • Whereas prior tool-using work commonly gives one agent access to all tools, flat planning can stop prematurely after tool errors; LLaMP instead uses hierarchical planning.The related-work examples include Coscientist and ChemCrow.Source: §3 · p. 3

4 Method

LLaMP combines supervisor-led orchestration of specialized ReAct agents with a response-consistency metric for numeric knowledge retrieval. The method separates tool handling from high-level task allocation and assesses response uncertainty and confidence.

4.1 Hierarchical orchestration

  • The supervisor allocates tasks among specialized assistants, a modular design intended to reduce individual agents’ cognitive load, context-window use, and schema parsing.The supervisor serves as a router and decision-maker and can coordinate complex tasks such as molecular dynamics simulations.Source: §4.1 · p. 3
  • Each assistant handles a specific tool or API schema and can refine calls from feedback; the two-level system equips assistants with distinct toolkits and data stores.The framework describes modular extension through additional assistant agents.Source: §4.1 · pp. 3–4, Figure 1 · p. 4

4.2 Self-consistency of response (SCoR)

  • For numeric retrieval in high-stakes workflows, the method defines consistency metrics that combine response uncertainty and the rate of valid responses.Precision uses the standard deviation of valid responses and their count; CoP maps precision to (0, 1], while confidence is the ratio of valid responses to trials.Source: §4.2 · p. 4
  • SCoR is intended to reflect reproducibility and practical usability: it reaches 1 for identical responses every time and 0 for high inconsistency or reluctance to answer.Figure 2 covers bulk moduli, formation energies, common-semiconductor bandgaps, and multi-element-material bandgaps; missing predictions are shaded and fliers circled.Source: §4.2 · p. 4, Figure 2 · p. 5

5 Experiments

LLaMP combines hierarchical ReAct agents with materials-data tools to handle multimodal queries, and it outperforms baselines on reported property-prediction benchmarks. The experiments also demonstrate literature-grounded synthesis retrieval, crystal editing, and language-driven simulation.

5.1 Multimodal ReAct Augmentation

  • Hierarchical ReAct agents decompose multimodal queries, delegate retrieval to specialist agents, and integrate materials data; flat planning struggles with tool use and RAG.For backbone models, reported function-calling capability ranks are Claude-3.5-Sonnet (#1), Gemini-1.5-Flash (#24), then Llama3-8B (#46).Source: §5.1 · p. 5, §5.2 · p. 7

5.2 Performance Benchmarks

  • LLaMP achieves the highest SCoR and lowest errors among evaluated models across bulk modulus, formation energy, and bandgap prediction.The comparison includes StructChem, Darwin, and vanilla LLMs; Table 1 reports averages over five runs with Materials Project values as ground truth.Source: §5.2 · p. 5, Table 1 · p. 6
  • LLaMP reduces bulk-modulus MAE from around 40 to 14.57 GPa, while vanilla LLM formation-energy SCoR and MAE range from 1.5 to 5.5 eV.Vanilla models sometimes rely on low-fidelity online values for bulk moduli; accurate formation energies require compound and elemental constituent energies.Source: §5.2 · p. 6
  • Vanilla LLMs handle common semiconductor bandgaps better than less-familiar multielement materials, whereas LLaMP retrieves multielement bandgaps with SCoR 0.938.The ablation reports the best performance with the complete MP tools; GGA-calculated bandgaps are noted to underestimate experimental values by 40–50%.Source: §5.2 · p. 6
  • On 800 materials, LLaMP with GPT-4 reaches 0.98 magnetic-ordering accuracy and 0.992 magnetization R2, while GPT-3.5 struggles with API fields and FM/FiM separation.GPT-3.5 may confuse total magnetization with magnetization per formula unit and omit normalization; GPT-4 separates ferromagnetic and ferrimagnetic classes.Source: §5.2 · p. 7, Table 2 · p. 7, Figure 3 · p. 7

5.3 Real-world Applications

  • MP-backed synthesis retrieval produces step-by-step procedures grounded in experimental papers, while vanilla GPT-3.5 gives irrelevant precursors or a procedure dissociated from its cited paper.For YMnO3, GPT-3.5 includes irrelevant lithium compounds and misses the lower applied energy of metathesis; its LiFePO4 procedure is hallucinated from pretraining.Source: §5.3 · p. 8
  • In crystal editing, GPT-3.5 distorts the cell despite placing Li tetrahedrally, while LLaMP’s retrieved MP structure anchors lattice parameters but places Li at a hexagonal interstitial.Both place Li at fractional coordinate (0.5, 0.5, 0.5); LLaMP’s Si bases follow the MP convention, whereas GPT-3.5’s cell shape and volume are highly distorted.Source: §5.3 · pp. 8–9, Figure 4 · p. 8, Table 3 · p. 8
  • With Python REPL, atomate2, and MACE-MP-0, LLaMP fetches a stable structure, builds a supercell, runs annealing MD from 300K to 800K and back, then plots temperature over time.Source: §5.3 · p. 9

6 Discussion

LLaMP uses a hierarchical ReAct agent graph whose assistants can recover from tool-input errors. Its effectiveness and scope remain bounded by the capabilities of its backbone LLM, available data, and supported workflows.

6 Discussion

  • Assistant ReAct agents can correct invalid tool inputs by using observations to refine their arguments after an error.The example describes MPThermoExpert entering a formula in the chemical-system field, then adapting its input after the observation step.Source: §6 · p. 9
  • LLaMP’s effectiveness depends on backbone LLM function-calling and reasoning, data quality and coverage, and the accuracy and scope of supported calculations.MP structures and magnetic configurations are incomplete, DFT can be insufficient in some cases, and current workflows cover only a few atomate2 MLFF and VASP calculations.Source: §6 · p. 9
  • The authors present LLaMP as a few-shot ReAct framework for extracting and manipulating materials informatics, with potential extension to more data sources and laboratory tools.They suggest that future work could explore its use in scientific hypothesis generation and data-driven experiments.Source: §6 · p. 9

A.1 List of Implemented Assistant Agents and Tools

The appendix lists modular assistants for retrieving Materials Project data and running simulations, alongside other tools. Its prompt template directs the supervisor through repeated reasoning, single-action tool calls, and observations before producing a final answer.

A.1 List of Implemented Assistant Agents and Tools

  • Materials Project assistants are modular, and the appendix says adding API endpoints is straightforward.Figure A.1 depicts a supervisor coordinating assistant ReAct agents, including thermodynamics and elasticity experts, and shows handling of API-calling errors.Source: §A.1 · p. 13, Figure A.1 · p. 14
  • The listed assistants cover Materials Project summaries, thermodynamic and elastic properties, magnetism, dielectric and piezoelectric data, electronic structure, synthesis, structures, and machine-learning-force-field simulations.The tool set also includes a Python REPL and arXiv and Wikipedia query tools.Source: §A.1 · p. 13

A.2 Prompt Template

  • The prompt template cycles through Thought, Action, and Observation, with each JSON tool call containing one action and the supervisor instructed to debug errors and refine assistant outputs.After the reasoning-and-action cycle, the template asks for a final answer to the original question.Source: §A.2 · p. 15

B Sample Questions

The sample-question appendix includes a QA comparison between LLaMP and GPT-3.5 and illustrates queries about material properties. The examples include bandgap requests and an elastic-tensor question, but the supplied passages do not establish comparative outcomes.

B Sample Questions

  • Table 5 is identified as a QA comparison between LLaMP and GPT-3.5, but the supplied table passage gives no comparative results.Source: Table 5 · p. 16
  • Another sample asks for NaCl’s elastic tensor; the accompanying response describes a 6x6 representation and gives approximate elastic constants.Source: B Sample Questions · p. 17
  • One sample asks for ground-state bandgaps across multiple materials, and the accompanying answer lists values with a bulk-material caveat.Source: B Sample Questions · pp. 17, 20

LLaMP GPT-3.5

The section contrasts a generic GPT-3.5 YMnO3 recipe with literature-grounded synthesis details, including a DOI-linked procedure. It also shows that the generated instructions can include questionable ingredients and inconsistent step numbering.

LLaMP GPT-3.5

  • The DOI-linked YMnO3 procedure specifies dissolving nitrate precursors with a hydroxide source, adding citric acid, and pre-calcining at 250°C for 2 hours.The cited DOI is 10.1021/ic502027k.Source: LLaMP GPT-3.5 · p. 21
  • The YMnO3 procedure gives a final heating range of 700°C to 930°C for 10 to 18 hours in air.This step follows pre-calcination.Source: LLaMP GPT-3.5 · p. 21
  • GPT-3.5's suggested precursor list includes lithium carbonate or lithium hydroxide, and its response begins to misnumber the procedure at Step 2.The listed materials also include yttrium oxide and manganese oxide.Source: LLaMP GPT-3.5 · p. 21

Question Answer

The Question Answer examples show retrieval of a stable diamond-cubic silicon structure and its modification by inserting lithium at a tetrahedral interstitial site. The resulting structure is returned as a JSON string.

Question Answer

  • The response returns the modified structure as a JSON string after the Li insertion.The output lists the inserted Li site alongside the Si sites.Source: Question Answer · p. 23
  • The retrieved stable Si structure is represented as a Pymatgen structure JSON with a lattice and atomic sites.The example gives the structure's space-group number as 227.Source: Question Answer · p. 23
  • The example retrieves the most stable diamond-cubic Si structure and inserts one Li atom at a tetrahedral interstitial site.The final structure includes both Si and Li sites.Source: Question Answer · p. 23

C.1 Automated Computational Workflow

The workflow example retrieves a LiTaO3 structure from Materials Project, builds a supercell, and runs molecular dynamics with MACE through atomate2 and jobflow. The run completes and the trajectory is used to plot temperature against time.

C.1 Automated Computational Workflow

  • The example uses atomate2 and jobflow with the MACE-MP-0 force field to run molecular dynamics.The requested settings specify NVT, a 300–800–300 K schedule, and a 2 fs timestep; the recorded tool call uses Langevin dynamics.Source: §C.1 · pp. 25, 27
  • The LiTaO3 structure is expanded into a 270-atom supercell, although the requested target was approximately 100 atoms.The original structure was retrieved from Materials Project.Source: §C.1 · pp. 25–27
  • The MACE molecular-dynamics workflow completed 1001 steps and reported a final energy of -2262.60595703125 eV.The run used a local jobflow execution.Source: §C.1 · pp. 27–28
  • The workflow reads the saved trajectory, extracts temperatures at successive steps, and plots temperature against time.The example saves the plot as temperature_vs_time.png.Source: §C.1 · p. 28

C.2 Simulation Code Generation and Execution

The section demonstrates LLaMP generating, correcting, and executing ASE code to calculate aluminum's bulk modulus. The corrected EMT calculation reports approximately 39.23 GPa.

C.2 Simulation Code Generation and Execution

  • LLaMP generates, self-corrects, and executes ASE simulation code to calculate the bulk modulus of aluminum.The calculation relaxes the structure, samples volumes and energies, and fits an equation of state.Source: §C.2 · pp. 29–30
  • The calculation uses a 2x2x2 Al supercell and the EMT calculator before fitting an equation of state to volume-energy values.The script evaluates scale factors from 0.98 through 1.02.Source: §C.2 · p. 29
  • 39.23 GPa is the reported bulk modulus for a relaxed 2x2x2 Al supercell calculated with the ASE EMT calculator.The initial output was described as extremely small because of a unit-conversion error; the corrected script converts eV/Angstrom^3 to GPa.Source: §C.2 · pp. 31–32
Loading 2401.17244v3…