Source-linked AI summary
LLM Agents Perform Controlled Experiments Using Simulation Models
Yuchen Xia, Michael Weyrich, Nasser Jazdi, Johannes Stümpfle, Johannes Sigel, Akshay Narla, Gavin K. Reynolds, Anna Jawor-Baczynska, Pol Llopart
TL;DR
Scientific and engineering applications need controlled experimentation to assess responses to intervention, beyond plausible language or code generation. This paper introduces a multi-agent framework that uses high-fidelity simulations for comparative experiments and reports more specific, evidence-grounded, and useful outputs, while treating the system as decision support rather than a replacement for user judgment.
Problem
Scientific and engineering tasks require reasoning about intervention effects through controlled comparison, which language-only systems do not naturally provide.
Method
A multi-agent system structures user tasks, formulates intervention hypotheses, runs comparative simulations, interprets outcomes, and synthesizes recommendations.
Results
Simulation integration produces more specific, evidence-grounded, and useful outputs for practical decision-making than language-only reasoning.
Takeaways & Limitations
Coupling LLM hypothesis generation with mechanistic simulation provides a minimal, generalizable approach to optimization-oriented reasoning.
Takeaways & Limitations
The system is intended to support, not replace, user judgment because perfect simulations are rarely achievable and LLMs lack precise training for every specific scenario.
Abstract
from arXiv · showhide
Large language models (LLMs) have shown strong capabilities in reasoning, planning, and tool use, but many scientific and engineering tasks require more than plausible text and code generation. They require understanding how a system responds to intervention, which in practice depends on controlled experimentation. In this work, we propose a multi-agent framework that enables LLM agents to conduct controlled experiments with scientific simulation models for pharmaceutical process design. Given a user query and a baseline configuration, the system constructs a structured task representation, designs experiments, executes comparative simulation, interprets the resulting outcomes, and synthesizes evidence-based recommendations for process parameter optimization. By coupling language models with high-fidelity simulation models in an interactive agent framework, the proposed system supports reasoning through intervention, comparison, and observation. As a result, it produces more specific and actionable outputs than language-only reasoning. In an industrial application setting, this advantage is reflected in higher output specificity as well as improved user-rated correctness and helpfulness. Ablation studies and visualized case analyses further demonstrate the effectiveness and practical utility of simulation-integrated experimental reasoning.
I. INTRODUCTION
Scientific and engineering tasks require controlled comparison to determine how systems respond to intervention, while existing LLM systems often produce suggestive rather than evidential outputs. The paper positions high-fidelity simulation as an experimental environment for testing hypotheses and comparing process modifications.
- Motivation: Controlled experimentation varies a selected factor while holding other conditions fixed, enabling specific, testable, and actionable conclusions.This logic is presented as central to scientific inquiry and engineering problem-solving.
- Research gap: Existing LLM-based systems commonly retrieve information or execute isolated functions rather than conduct structured experimental comparisons.Their outputs may therefore remain suggestive when reliable conclusions depend on controlled intervention.
- Application setting: Pharmaceutical simulation models provide a practical basis for evaluating candidate process modifications and their consequences under varying operating conditions.The models capture system dynamics relevant to industrial process design.
- Proposed direction: The proposed perspective treats simulation as an environment where hypotheses are tested through controlled intervention and comparison, rather than as a passive auxiliary tool.This perspective motivates the framework developed in the work.
- Related work: Prior LLM-science systems emphasize retrieval, workflow automation, code generation, or simulation setup instead of using simulation for comparative hypothesis testing.The paper identifies this distinction as the gap addressed by its framework.
C. Simulator and tool integration
The framework connects LLM agents to high-fidelity scientific simulation through a structured, role-based workflow. It decomposes user queries into requirements and plans, executes parameterized experiments, interprets outputs, and exposes intermediate artifacts for industrial observability.
- Scientific reasoning as structure: The architecture organizes reasoning as a scientific workflow that decomposes tasks, designs controlled experiments, observes outcomes, and synthesizes conclusions.The stated structure is intended to support systematic and verifiable engineering reasoning.
- Scientific simulation: Unlike simplified or sandbox environments, the framework targets high-fidelity scientific simulations that reproduce dynamic physical and chemical phenomena.This allows complementary knowledge from the simulation model and language model to inform reasoning.
- Application-driven design: A directed graph formalizes intermediate reasoning artifacts, providing observability and visualization for practical industrial use.The graph represents artifacts generated and transformed during the workflow.
- Information flow: The system transforms a user query into explicit intermediate artifacts, simulation-based evidence, and final recommendations.This transformation is distributed across specialized agents and their information flow.
- Agent architecture: The six-agent pipeline includes five LLM-driven agents and a rule-based Executor, with distinct roles for requirements, planning, execution, interpretation, and reporting.The Requirement Analyzer structures the task, the Planner creates an abstract plan, and downstream agents operationalize, interpret, and summarize it.
B. Interactive Interface Between LLM Agents and the Simulation Model
The simulation interface uses an Operator–Executor dialogue loop with an Interpreter that analyzes visual outputs. The evaluation compares full and ablated configurations to isolate the effects of simulation integration, requirement analysis, and the multi-agent workflow.
- Interactive interface: The Operator, Executor, and Interpreter coordinate simulation invocation through a two-agent execution loop and visual-output analysis.The Operator and Executor exchange calls and results, while the Interpreter analyzes plots.
- Interactive interface: The Operator converts abstract plan steps into executable controlled simulation experiments by generating fully specified function-call code.It resumes reasoning after receiving execution results from the Executor.
- Interactive interface: The deterministic Executor runs modular simulation functions and returns textual and graphical outputs for downstream reasoning and interpretation.Its role is reliable tool execution rather than language reasoning.
- Interactive interface: If a function call fails, the Operator can revise and retry it until execution succeeds or a predefined retry limit is reached.This protocol is described as supporting adaptive and fault-tolerant tool use.
- Evaluation design: The evaluation compares the full system with No Simulation, No Requirements, and LLM-only configurations.These variants isolate simulation integration, requirement structuring, and the multi-agent workflow.
B. Tasks
The tasks ask the system to improve pharmaceutical crystallization processes from user-defined baselines by controlled parameter perturbation and simulation comparison. The workflow records outputs, reasoning trajectories, plots, and technical and user-based evaluation data.
- Tasks: Each task specifies a user-defined baseline experiment configuration together with a task goal in pharmaceutical crystallization process design.The test set contains five task scenarios.
- Task formulation: The system represents the baseline process as a parameter vector x and seeks an improved outcome for an optimization goal g.The goal may include maximizing yield.
- Experimental procedure: For each relevant baseline parameter, the system generates a controlled perturbation while keeping all other parameters unchanged.The perturbed vector is written as x′ = (x(1), . . . , x(i)′, . . . , x(n)).
- Experimental procedure: The simulation compares baseline f(x) with perturbed f(x′), after which the system summarizes findings and recommends x∗ closer to g.This sequence follows hypothesize → intervene → observe → analyze → report.
- Simulation model: The proprietary crystallization model combines first-principles equations with calibrated semi-empirical formulations to reproduce mass transfer, phase transitions, and evolving material distributions.The model consists of differential and algebraic equations and is exposed through callable software interfaces.
- Data collection: For each system variant and task, the study collects final outputs, intermediate reasoning, simulation plots, reasoning graphs, and technical and user-annotation evaluation data.The technical metrics quantify linguistic uncertainty, while user annotations assess correctness and practical usefulness.
V. RESULTS EVALUATION
The evaluation compares simulation-integrated and language-only reasoning, showing that simulation produces more specific, evidence-grounded, and actionable process recommendations. The analysis also measures reasoning quality and simulation-calling behavior.
- Qualitative Result Comparison: Simulation integration shifts responses from vague heuristics to specific, evidence-based recommendations grounded in comparative simulation results.The representative crystallization task contrasts a cautious language-only suggestion with a simulation-supported recommendation tied to cooling-rate effects.
- Qualitative Result Comparison: The full system compares cooling rates and links higher cooling rates to larger median particle sizes, narrower PSD width, higher supersaturation, and larger MZW.It recommends increasing the cooling rate beyond 5.25 to target a median particle size of 50 µm.
- Evaluation Perspectives: Reasoning quality is evaluated through specificity, correctness, helpfulness, and simulation-calling behavior, including simulation-call precision, recall, and hypothesis validation.The evaluation combines linguistic metrics with user-oriented and agent-behavior measures.
2) Metrics for Reasoning Correctness and Usefulness:
The study evaluates correctness and usefulness through domain-specialist ratings while also measuring simulation-call quality and hypothesis confirmation. Across five crystallization tasks, the Full System outperforms ablated variants on specificity, correctness, and helpfulness.
- Correctness and Usefulness: Two senior domain specialists rate system outputs for correctness and practical helpfulness across five pharmaceutical crystallization scenarios.Correctness and helpfulness use 1–5 scales and are averaged over the five task scenarios.
- Simulation-Call Metrics: Simulation-call precision measures appropriate calls, recall measures necessary calls made, and simulation confirmation measures hypotheses validated by simulation outputs.A hypothesis is confirmed when simulation results agree with its prediction and move the system closer to the optimization goal.
- Quantitative Results: The Full System outperforms all ablated variants in output specificity, correctness, and helpfulness.Its responses contain the fewest vague expressions, with 13.7 per 1,000 words and a LUCI score of 0.13; No Simulation and LLM-only have LUCI scores of 0.33 and 0.36.
- Quantitative Results: 94% simulation-call precision and 76% hypothesis support indicate that the Full System frequently executes appropriate experiments whose outputs validate its reasoning.The process forms a hypothesis → experiment execution → validation loop through perturbed simulation inputs.
- Quantitative Results: Domain specialists assign the Full System an average correctness score of 4.1 and helpfulness score of 4.2.These ratings summarize expert evaluation of the system outputs in the reported task set.
E. Ablation on Requirement Analysis (No Requirement)
Removing simulation access produces vague, unverifiable outputs, while requirement analysis and simulation integration support more targeted, evidence-grounded reasoning for optimization tasks.
- Requirement Analysis Ablation: Removing requirement analysis increases simulation recall to 69% but reduces simulation precision to 65%.The system overuses available simulation functions and produces less targeted calls without explicit requirement reasoning.
- Simulation Ablation: Simulation removal yields the weakest performance, with outputs too vague for meaningful insight extraction.Domain users regarded fuzziness and LUCI scores below 3.0 as unusable in this setting.
- Simulation Ablation: The LLM-only baseline produces overly vague, hedged responses that engineering users consider unverifiable and unhelpful.
- System Design: The agent workflow analyzes problems, formulates intervention hypotheses, compares simulation outcomes, and incorporates evidence into conclusions.This process follows the scientific sequence of hypothesize, intervene, observe, and report.
- Overall Findings: Simulation integration makes outputs more specific, evidence-grounded, and useful for practical decision-making.The framework combines LLM hypothesis generation with scientific simulation in optimization-oriented reasoning.
APPENDIX: REASONING TRAJECTORIES
The reasoning trajectories document controlled process experiments, practical trade-offs, implementation planning, and monitoring criteria for crystal-growth optimization.
- Supersaturation & MZW: Higher supersaturation with longer cooling duration provides more driving force for growth, while nucleation remains important to control.The trajectory proposes adjusting cooling rates or adding additives to regulate nucleation and growth.
- Controlled Experiment: Increasing cooling duration from 12.0 to 18.0 hours produces slightly larger and more uniform crystals with higher supersaturation.Further modifications are needed to reach a median crystal size of 50 µm.
- Trade-offs: Candidate modifications include changing cooling rate and duration and reducing seed loading, with trade-offs for process time, nucleation control, crystal-size variability, yield, and purity.
- Implementation: Implementation planning calls for adjusting parameters according to simulation insights, monitoring crystal-size variance, and documenting operational changes.
- Evaluation Criteria: Success criteria include reaching a crystal median size of at least 50 µm while maintaining product quality and process efficiency.
- Process Control: Real-time monitoring of crystal size and supersaturation supports dynamic adjustment of cooling rates and durations during process operation.
Task Sample 2 (Full System)
Controlled experiments compare crystallization outcomes under targeted parameter changes. The full-system results identify higher seed loading, faster cooling, and higher crude concentration as actionable adjustments for crystallization performance, yield, or particle-size control.
- Seed Loading: Increasing seed loading from 0.10 to 0.15 accelerates crystallization and improves yield, while producing smaller particles and a narrower particle size distribution.The API solid-phase mass reaches approximately 0.0038 kg at 20 hours versus about 0.0035 kg for seed_loading = 0.10.
- Initial Temperature: Lowering the initial temperature to 63.0°C is associated with faster crystallization, higher yield, faster d50 recovery, and lower stabilized PSD width.These outcomes are evidenced by more rapid API solid-phase mass increase and a higher final value.
- Crude Concentration: Increasing crude concentration from 35 to 45 accelerates crystallization and increases yield.At concentration 45, API solid-phase mass rises faster and reaches approximately 0.0045 kg versus 0.0035 kg for concentration 35.
Task Sample 3 (Full System)
The controlled experiments compare crystallization settings across antisolvent volume, seed loading, and cooling rate. Results reveal trade-offs between crystallization speed, solid mass, particle size distribution, and desired-polymorph stability.
- Cooling rate 5.25 had minimal impact on PSD width, with both conditions stabilizing around 30 μm.
- Cooling rate 5.25 produced slightly higher API solid-phase mass, larger d50, and greater supersaturation and MZW than cooling rate 4.0.
- Antisolvent volume 0.95 yielded slightly higher final solid mass and larger d50, while particle-size trends remained similar between 0.95 and 0.90.
- The experiments recommend antisolvent volume 0.95, seed loading 0.10, and cooling rate 5.25 when prioritizing polymorph stability and process control.
- Seed loading 0.15 increased crystallization speed and final solid mass, but seed loading 0.10 produced larger particles, narrower PSD, and lower supersaturation.
Task Sample 4 (Full System)
The system recommends monitoring crystallization in real time by tracking temperature, concentration, crystal size distribution, and process efficiency. Deviations should prompt timely process adjustments.
- The monitoring step is intended to ensure desired outcomes while allowing crystallization conditions to be adjusted as necessary.
- Monitoring tracks temperature, concentration, crystal size distribution, and process efficiency to assess whether desired outcomes are achieved.
- Deviations from expected behavior should be addressed promptly through real-time process adjustments.
Task Sample 5 (Full System)
The final task evaluates how to implement and validate optimized crystallization modifications. The workflow selects modifications based on experimental results, updates the protocol, and uses repeated trials to assess consistency and reliability.
- Higher cooling rate 7.0 produced faster crystallization, with quicker API solid-phase mass increase and larger final particle size.
- Modified setups are compared with the original protocol to assess reduced crystallization time while maintaining or improving crystal quality.
- The workflow selects the most effective modifications based on experimental results and their practical effect on crystallization time and quality.
- The chosen parameter values and procedural adjustments are documented in an updated crystallization protocol.
- Repeated trials validate whether optimized parameters consistently deliver faster crystallization with satisfactory crystal quality.