Source-linked AI summary
Evaluating LLM-based AI agents integrated with materials synthesis tools: the case of atomic layer deposition
Angel Yanguas-Gil
TL;DR
Materials-synthesis applications lack sufficiently complete, technique-specific evidence about LLM-agent performance, despite growing integration with experimental tools. This review surveys agent technologies and evaluation strategies, uses ALD as a case study, and proposes a practical framework spanning knowledge, reasoning, tool use, synthesis challenges, and closed-loop interaction. It concludes that robust evaluation must address performance, robustness, reproducibility, and safety across these facets.
Problem
Materials synthesis has few technique-specific benchmarks, and existing coverage is often too limited to provide a complete picture of LLM-agent performance.
Method
The review synthesizes evaluation strategies for LLM agents integrated with materials-synthesis tools, using ALD and virtual or experimental interactions as a case study.
Results
The review presents a multidimensional evaluation framework spanning knowledge and reasoning, tool use, synthesis-specific challenges, and closed-loop interaction with experimental or virtual tools.
Takeaways & Limitations
Technique- and tool-specific benchmarks provide a pathway for evaluating agent designs and iterating them without waiting for controlled experiments.
Takeaways & Limitations
Agent performance depends on the LLM, generation settings, prompting, accessible tools and information, and harness design; stochastic outputs also create run-to-run variability.
Abstract
from arXiv · showhide
This work provides an overview of the different strategies that can be used to evaluate the performance of AI models and agents based on large language models (LLMs) for materials synthesis. After providing a brief overview of the key technologies behind the current generation of AI agents based on LLMs, we summarize the different approaches to evaluating these models in the context of materials science and in particular on materials synthesis, with a specific emphasis on scenarios in which the models are directly integrated with experimental tools. We discuss evaluation strategies spanning knowledge and reasoning benchmarks, tool-use benchmarks, and closed loop benchmarks involving the interaction with experimental systems or realistic virtual tools. We use atomic layer deposition (ALD) as a case study, emphasizing how existing approaches in the literature both build from general approaches used beyond materials science and can be generalized to other materials synthesis techniques. Finally, we provide a practical evaluation framework to evaluate LLMs in the context of materials synthesis
1 Introduction
Materials-science AI efforts are moving toward LLM agents that orchestrate complex workflows and augment existing experimental tools. This creates a need for domain-relevant evaluation strategies that quantify performance, expose limitations, and reduce risks before deployment.
- LLM-based agents could orchestrate workflows spanning synthesis, characterization, simulation tools, databases, and prior data.
- AI integration with experiments includes both end-to-end autonomous platforms and agents augmenting existing instruments such as AFM, pulsed laser deposition, and ALD.
- Evaluating agents is critical for understanding research-task performance, identifying limitations, and quantifying the value of AI integration.
- Evaluation is especially important in settings such as semiconductor processing, where contamination or low yields can propagate downstream across processing steps.
- Targeted benchmarks developed jointly by AI researchers and materials scientists can evaluate models in scenarios relevant to downstream materials-research uses.
- The review surveys experimental-tool integration and evaluation strategies, using ALD as a model system and concluding with a practical materials-synthesis evaluation framework.
2 LLM-based agents and their integration with experiments
LLM-based agents combine probabilistic language models with harnesses, reasoning strategies, and tool interfaces to perform multistep tasks. In experimental settings, structured model outputs and iterative harnesses connect agents to instruments through software APIs.
- 2.1 Overview of LLMs, multimodal LLMs, and AI agents: LLMs generate stochastic outputs, so identical prompts can produce different continuations, affecting reproducibility and consistency in scientific applications.
- 2.1 Overview of LLMs, multimodal LLMs, and AI agents: A harness is an orchestration layer that manages interaction between an LLM and its user or environment.
- 2.1 Overview of LLMs, multimodal LLMs, and AI agents: LLM agents combine multistep reasoning and tool use to interact with environments and pursue concrete goals without task-specific programming.
- 2.2 Integrating LLM-based agents with experimental tools: Coscientist uses an LLM planner with Google, Python, documentation, and experiment commands, while a non-LLM automation layer executes calls and returns results.
- 2.2 Integrating LLM-based agents with experimental tools: The ALD integration places a translation layer between a Python API and LabVIEW backend, supporting tool use, MCP, and a custom harness.
- 2.2 Integrating LLM-based agents with experimental tools: Experimental integration commonly uses structured model outputs, a harness that translates them into executable actions, and an iterative control loop.
- 2.2 Integrating LLM-based agents with experimental tools: A simple experimental agent repeatedly queries the LLM, translates responses into tool instructions, and stops when the task is complete or an iteration limit is reached.
3 Evaluating LLMs and LLM-based agents in materials science
Evaluation strategies for LLMs in scientific applications and materials science include methods that assess knowledge, reasoning, tool use, and interaction with experimental or virtual environments. The review then applies these strategies specifically to materials synthesis through an ALD case study.
- The review surveys evaluation strategies for LLMs in scientific applications and materials science before focusing on materials synthesis through ALD.
3.1 General evaluation strategies and benchmarks in materials science
Materials-science evaluation uses question-based knowledge and reasoning benchmarks, tool-use tests, and interactive environments, but materials synthesis remains sparsely covered. The review therefore identifies substantial gaps in assessing LLMs and agents for synthesis.
- Knowledge, reasoning, and text generation: Knowledge and reasoning benchmarks use multiple-choice, open-response, numerical, formatted-text, code, and multimodal questions with scoring procedures.Open-response examples include physical-property retrieval, numerical problems, chemical formulas, SMILES, and code generation.
- Knowledge, reasoning, and text generation: Most benchmarks share a dataset, an input builder, and an evaluator that parses model outputs and computes scores, with metadata optionally guiding evaluation.The pipeline can include question difficulty, subcategory, or additional context as metadata.
- Tool use and interactive agent behavior: Tool-use evaluation distinguishes single-step API-call generation from multi-step interaction in which agents receive environmental feedback until reaching a desired outcome.Single-step accuracy can compare generated calls with a desired call without physically connecting the model to instruments.
- Tool use and interactive agent behavior: Scientific-tool benchmarks can treat the agent as a black box communicating through an interface, making interface design central to tool-specific evaluation.The interface is described as the natural connection point between experiments and AI models.
- Tool use and interactive agent behavior: Multi-step evaluation requires a tool or environment, but virtual tools can approximate real systems and provide insight into performance before physical deployment.The review presents idealized virtual systems as possible substitutes for real tools in some cases.
- Benchmarks for materials synthesis: Materials synthesis is underrepresented: broad benchmarks emphasize metallurgy, while technique-specific benchmarks remain few, including a 20-question graphene-synthesis benchmark.The review concludes that the literature has significant gaps in understanding LLM and agent capabilities and limitations for materials synthesis.
3.2 Atomic layer deposition as a case study
Atomic layer deposition (ALD) provides a model system for evaluating LLM-based agents across knowledge, tool-use, and closed-loop process-optimization tasks. The case study shows that evaluation must be tailored to experimental tools and interfaces while accounting for task difficulty, process specificity, and stochastic agent behavior.
- Atomic layer deposition as a case study: ALD supports evaluation at multiple levels, from identifying chemistries and selecting precursors to closed-loop process optimization.Its high degree of automation makes it suitable for integration with conventional machine learning and LLM-based agents.
- Open-response evaluation: ALDBench uses 70 expert-graded questions to evaluate ALD knowledge and reasoning across growth, process, general-knowledge, and application categories.Seven experts scored response quality, specificity, relevance, and accuracy on 1–5 Likert scales, while questions were rated for difficulty and specificity.
- Open-response evaluation: Higher question difficulty was associated with lower response quality and relevance, while greater question specificity was associated with lower response accuracy.These three relationships were statistically significant according to Fisher exact tests with p-values < 0.05.
- Tool-use evaluation: Tool-use benchmarks distinguish translating direct instructions into experimental function calls from identifying the processes needed to achieve a requested outcome.Agents generally performed better on instruction challenges than on process-discovery challenges, which required inferring precursors and channels.
- Tool-use evaluation: Process-identification performance varied with the chemistry and literature coverage of the challenge, including poorer performance on less-represented and counterintuitive processes.Because these results depend on the agent type and interface, benchmarks should be specific to each experimental tool and cover diverse conditions.
- Closed-loop optimization: Closed-loop optimization showed near-saturated growth per cycle but substantial run-to-run variation in dose times and search strategies across ten independent runs.The evaluation tracked requested experiments and iterations, revealing stochasticity in both optimization outcomes and environmental interaction.
4 Discussion and practical recommendations
Materials-synthesis agents remain incompletely evaluated, despite established evaluation strategies that can be adapted to the field. The proposed framework combines expert review, synthesis challenges, and closed-loop interaction benchmarks while accounting for stochasticity, robustness, safety, and implementation-dependent performance.
- Evaluation gaps: Materials-synthesis benchmarks currently provide incomplete coverage, including for atomic layer deposition, leaving major gaps in understanding model capabilities and limitations.The review notes that only a few techniques have specific benchmarks and that ALD benchmarks remain incomplete.
- Practical evaluation framework: The framework begins with independently answered, expert-reviewed open-response questions assessing knowledge and abilities around a specific synthesis technique.Domain experts can review response quality, with independence between questions treated as a key methodological consideration.
- Practical evaluation framework: Synthesis-challenge benchmarks assess both execution of specific instructions and inference of instructions from a materials-synthesis case.Process-identification challenges for ALD are given as an example.
- Practical evaluation framework: Closed-loop multistep benchmarks use virtual tools, simulations, or experimental-system models to provide feedback until an agent meets a specified requirement.The virtual tools should sample a sufficiently broad range of behaviors rather than represent only one process or instrument.
- Gaps, limitations, and opportunities: Evaluation must address stochastic run-to-run variation, sensitivity to phrasing, and safe failure behaviors such as interruption, clarification requests, and avoiding harmful actions.These concerns are especially relevant when agents are integrated with experimental systems.
- Gaps, limitations, and opportunities: Benchmark results depend on the model, generation settings, prompts, accessible tools and information, and harness design, requiring repeated evaluation as agents evolve.Technique- and tool-specific benchmarks can support rapid design iteration without compromising safety or waiting for controlled experiments.
5 Conclusions
Integrating LLM agents with materials-synthesis tools creates opportunities for autonomous experimentation while raising questions about performance, robustness, reproducibility, and safety. The review therefore advocates multifaceted evaluation and uses ALD to illustrate evaluation from process identification through optimization, alongside remaining limitations in inference, robustness, and variability.
- Conclusions: Robust evaluation of materials-synthesis agents must span knowledge and reasoning, tool use, synthesis-specific challenges, and closed-loop experimental or virtual interactions.The conclusion frames these as multiple necessary facets of evaluation.
- Conclusions: Atomic layer deposition supports evaluation at multiple levels because its high degree of automation enables process identification, tool-use, and full process-optimization studies.The case study illustrates how one automated synthesis technique can support progressively broader evaluations.
- Conclusions: The ALD examples show promise alongside limitations in inferring new processes, robustness, and run-to-run variability.The conclusion identifies these limitations as central considerations for current LLM-based agents.
- Conclusions: More comprehensive, synthesis-specific benchmarks are needed to support safe integration of these systems into research workflows and development of models tailored to materials synthesis.This recommendation is stated within the scope of the ALD case study and the broader review.