PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery

Pingchuan Ma, Tsun-Hsuan Wang, Minghao Guo, Zhiqing Sun, Joshua B. Tenenbaum, Daniela Rus, Chuang Gan, Wojciech Matusik

arXiv:2405.09783v1cs.LGcs.AIcs.CE

TL;DR

LLM-driven scientific discovery is limited in its ability to calculate physical outcomes, while research methods are often tailored to individual disciplines. SGA combines LLM hypothesis generation with simulation-based feedback and differentiable optimization, and experiments report strong outcomes in constitutive-law discovery and molecular design, including unconventional solutions deemed reasonable by domain experts.

  • Problem

    LLM-driven approaches are limited by their computational capability for physical-science tasks, while research streams often develop discipline-specific methods rather than a unified approach.

    Source: §1 · pp. 1–2

  • Method

    SGA uses LLMs to generate and revise scientific hypotheses while simulations provide observational feedback and differentiably optimize continuous parameters.

    Source: §1 · pp. 2–3

  • Results

    SGA outperforms other LLM-based benchmark methods in constitutive-law search and molecular design, with unconventional solutions judged reasonable by domain experts.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The framework produces novel molecules and constitutive laws that differ from conventional expectations yet remain sensible under expert analysis.

    Source: §1 · p. 3

    The authors note that directly executing LLM-generated code without filtering creates potential AI safety risks to the operating system.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large Language Models have recently gained significant attention in scientific discovery for their extensive knowledge and advanced reasoning capabilities. However, they encounter challenges in effectively simulating observational feedback and grounding it with language to propel advancements in physical scientific discovery. Conversely, human scientists undertake scientific discovery by formulating hypotheses, conducting experiments, and revising theories through observational analysis. Inspired by this, we propose to enhance the knowledge-driven, abstract reasoning abilities of LLMs with the computational strength of simulations. We introduce Scientific Generative Agent (SGA), a bilevel optimization framework: LLMs act as knowledgeable and versatile thinkers, proposing scientific hypotheses and reason about discrete components, such as physics equations or molecule structures; meanwhile, simulations function as experimental platforms, providing observational feedback and optimizing via differentiability for continuous parts, such as physical parameters. We conduct extensive experiments to demonstrate our framework's efficacy in constitutive law discovery and molecular design, unveiling novel solutions that differ from conventional human expectations yet remain coherent upon analysis.

1. Introduction

Physical-science discovery lacks a unified approach across disciplines, and LLM-driven methods face limits in computational capability. SGA combines LLM hypothesis reasoning with physical simulation in a bilevel framework, evaluated on constitutive-law discovery and molecular design.

  • The framework aims to unify physical-science discovery across disciplines, where research streams often develop methods tailored to individual fields.The paper links a unified approach to democratizing access to research tools, standardizing practices, and improving efficiency across disciplines.Source: §1 · p. 1
  • SGA assigns hypothesis generation and revision to LLMs, while simulations provide observational feedback and differentiably optimize continuous parameters.The LLM reasons about discrete components such as equations or molecule structures; simulations optimize continuous components such as physical parameters.Source: §1 · p. 2
  • The empirical study targets molecular design from molecular structure, conformation, and quantum-mechanical properties, and constitutive-law discovery from recorded motion trajectories.For constitutive-law discovery, the method identifies symbolic equations and corresponding mechanical properties.Source: §1 · pp. 2–3
  • SGA discovers molecules and constitutive laws, outperforms other LLM-based baselines, and produces unconventional solutions judged reasonable by domain experts.The experiments cover physics-law discovery and molecular design.Source: §1 · p. 3

2. Scientific Generative Agent

SGA places LLMs at the upper level to propose scientific solutions and simulations at the lower level to validate them. The pipeline turns simulation outcomes into feedback for revising discrete expressions and optimizing continuous parameters.

2. Scientific Generative Agent

  • LLMs propose scientific solutions at the upper level, while simulations serve as experimental platforms for lower-level validation.Source: §2 · p. 3

2.1. Bilevel Optimization Pipeline

  • The simulator Φ maps an expression E and continuous components θ to a physical phenomenon y and observational feedback z.The outer level chooses E and the inner search space Θ; invalid expressions that cannot be simulated are excluded.Source: §2.1 · p. 3

2.2. LLM-Driven Outer-Level Search

  • The LLM analyzes simulation results, plans and proposes executable hypotheses, and generates offspring from several high-performing candidates.This selection supports simulation feasibility and allows hypotheses to combine ideas from different past experiments.Source: §2.2 · p. 4
  • Equation search proposes both an expression E and parameterization Θ, while entity search proposes structures E and uses simulation feedback z to inform revisions.Expert-crafted functions extract compact, relevant observations for the LLM.Source: §2.2 · p. 4
  • The exploit-and-explore strategy uses different LLM decoding temperatures for cautious and riskier proposals; the authors report a 1:3 exploit-to-explore ratio as a rule of thumb.Exploratory proposals can be too random or invalid, while exploitative proposals may repeat earlier solutions.Source: §2.2 · p. 4

2.3. Differentiable Inner-Level Optimization

  • Differentiable simulation uses gradient-based optimization to tune continuous parameters θ within Θ and returns loss curves or other recordings as feedback for revision.The feedback can include auxiliary recordings such as velocities across optimization iterations.Source: §2.3 · p. 4

3. Experiments

Experiments evaluate SGA across constitutive-law discovery and molecular design, comparing it with LLM and domain-specific baselines. SGA generally performs better, with ablations examining bilevel optimization, model choice, and exploration.

3.1. Problem Definitions

  • Constitutive-law discovery fits symbolic material models and continuous parameters to particle trajectories, while molecular design targets structures and 3D conformations matching quantum properties.Molecular structures are represented as SMILES strings, with atom coordinates as continuous parameters.Source: §3.1 · pp. 4–5

3.2. Experiment Setup

  • Evaluation covers four constitutive-law tasks and four molecular-property tasks, using five random seeds and comparisons against LLM-based baselines and method variants.The molecular targets include HOMO, LUMO, and HOMO-LUMO energy-gap values.Source: §3.2 · pp. 5–6, §3.3 · p. 6

3.3. Physical Scientific Discovery

  • Across eight tasks, SGA significantly outperforms four LLM baselines; it also tops symbolic regression on simplified task (a) and has lower loss than GhemGE on molecule tasks.The symbolic-regression comparison simplifies task (a) by removing back-propagation through time and using direct stress-tensor supervision.Source: §3.3 · pp. 6–7

3.4. Ablation Study

  • On an invented law mixing von Mises plasticity, granular material, and weakly compressible fluid, SGA still discovers the law with low quantitative loss.The authors report little visual difference between the ground-truth material and the optimized law.Source: §3.4 · p. 7
  • Bilevel optimization achieves better loss than outer-level LLM optimization alone, and LLM proposals and simulation-optimized results improve one another across steps.The authors characterize LLMs as cross-discipline generalists and simulations as domain specialists.Source: §3.4 · p. 7
  • GPT-4 statistically outperforms the other tested backbones overall, while Claude-3-Sonnet ranks second on most constitutive-law tasks and Mixtral-8x7B tops two molecule tasks.The study also tests GPT-3.5.Source: §3.4 · p. 7
  • Removing exploitation sharply increases simulation failures, while removing exploration worsens optimization despite high success rates, especially in nonlinear, unstructured searches with many local optima.Source: §3.4 · p. 7

3.5. Case Study

  • Case studies produce a quadratic deviatoric term from a linear material and distinct gap-targeted molecules: gap-0 uses sulfur, chlorine, and trifluoromethyl, while gap-2 has oxygen and sulfur rings.The constitutive-law implementation also supplies an init function that defines continuous parameters for inner-level optimization.Source: §3.5 · p. 8

4. Related Work

Related work spans automated scientific discovery, increasingly capable LLM agents, and bilevel optimization; SGA is situated against domain-specific automation and manually bounded search spaces.

4.1. Automated Scientific Discovery

  • Automated discovery links hypothesis formation with experimental data analysis, with constitutive-model identification and molecular design as example application areas.Prior constitutive-model work frames identification as system identification using differentiable simulation; machine learning also supports molecular prediction and structure optimization.Source: §4.1 · p. 8

4.2. Large Language Models and Agents

  • LLM-agent work includes tool creation and gradient-free optimization, while FunSearch and AlphaGeometry illustrate scientific and mathematical problem solving.The cited examples include FunSearch paired with evaluators for combinatorics and bin packing, and AlphaGeometry solving olympiad-level geometry problems without human demonstrations.Source: §4.2 · p. 9

4.3. Bilevel Optimization

  • Bilevel optimization nests an upper-level problem over a lower-level one; neural architecture search similarly optimizes discrete architectures above continuous weights.Typical NAS methods require predefined search spaces, whereas this framework uses LLM general knowledge to avoid manually specified discrete-search boundaries.Source: §4.3 · p. 9

5. Conclusion

The conclusion presents SGA as a framework pairing LLM-proposed scientific solutions with simulation feedback and reports stronger benchmark performance and expert-reviewed unconventional solutions. The appendix prompts specify iterative analysis and distinct representations for material laws and molecules.

5. Conclusion

  • The authors flag limits in solution interpretability, unfiltered generated-code safety, reliance on LLM internal knowledge, differentiability, and inference expense.They report about $10 per GPT-4 task and note that costs grow concerning as iterations increase.Source: §5 · p. 9
  • SGA outperforms other LLM-based benchmarks on constitutive-law search and molecular design, with consistent, robust, nearly monotonic improvement and expert-reviewed unconventional solutions.The conclusion also reports identifying unknown true constitutive laws and molecular structures.Source: §5 · p. 9

A. Full Prompts

  • Constitutive-law prompts require iteration-by-iteration feedback analysis, a detailed plan separating symbolic laws from continuous parameters, and commented code.For elastic-law code, continuous parameters go in __init__ and symbolic constitutive code in forward.Source: A. Full Prompts · pp. 14–15
  • Molecule-design prompts separate a discrete SMILES structure from 3D atom coordinates and require a valid structure with at least eight atoms.The coordinate matrix must have shape (N, 3), match the atom count, and contain distinct rows.Source: A. Full Prompts · p. 16

B.1. Data Workflow

The data workflow builds the LLM input from system instructions, prior solution-and-feedback history, and output-format constraints in a prescribed order.

B.1. Data Workflow

  • The prompt combines system instructions, iteration information, and a format prompt; the authors say ordering matters for performance and successful parsing.The system prompt is placed first or supplied as a special instruction depending on the LLM.Source: §B.1 · p. 16
  • Iteration history pairs each solution's code with its feedback and stacks the top-K solutions in reverse order.The resulting prompt appends the format prompt after the system prompt and iteration entries.Source: §B.1 · p. 16
  • The full prompt is formed by joining the assembled system, history, and format entries with newline separators.The format prompt is appended after the iteration loop.Source: §B.1 · p. 16

B.2. Differences to Symbolic Regression Task

The paper frames its task as loss-guided discovery under sparse direct supervision, then emphasizes code-based flexibility across input-output structures and disciplines.

B.2. Differences to Symbolic Regression Task

  • The task optimizes a generated solution through a potentially complex simulator, rather than directly fitting input-output pairs, to reflect sparse direct supervision.The formulation is presented as a superset of regular regression because the simulator can be constructed to cover that case.Source: §B.2 · p. 16
  • Code-based representation supports arbitrary input variables and output features, including multi-dimensional arrays and tensor operations.The authors say this avoids symbolic-regression input limits and preserves structural correlations among output dimensions.Source: §B.2 · pp. 16–17
  • The method is designed to adapt across disciplines through natural-language guidance, using LLMs pretrained on multidisciplinary text, mathematics, and code.The paper contrasts this customization with domain-specific hard-coded constraints and heuristics in traditional symbolic regression.Source: §B.2 · p. 17

C.1. Symbolic Regression

The section points to a full symbolic-regression comparison in Table 5 but supplies no comparison outcomes.

C.1. Symbolic Regression

  • The full comparison with symbolic regression methods is presented in Table 5.The supplied passages do not report the table’s values or identify a winning method.Source: §C.1 · p. 17, Table 5 · p. 17

C.2. Longer Iteration

The study examines longer runs of 20 question-answering iterations and finds iteration count especially consequential for the most challenging tasks.

C.2. Longer Iteration

  • Longer runs of 20 iterations are evaluated in a repeat of the Table 1 experiment, with results reported in Table 6.The study varies the number of question-answering cycles to assess its effect and provide practitioners with hyperparameter guidance.Source: §C.2 · p. 17, Table 6 · p. 17
  • Iteration count is a determining hyperparameter: it has little effect on easier tasks but markedly improves the most challenging tasks, including (b) and (c).The supplied passages provide no numerical results from Table 6.Source: §C.2 · pp. 17–18
  • Practitioners adapting the method should consider the number of iterations first.This recommendation follows the reported contrast between easier and more challenging tasks.Source: §C.2 · p. 18

D.1. Constitutive Law Discovery (a)

The appendix presents optimized implementations for constitutive-law tasks, an imaginary constitutive law, and molecular-design tasks.

D.1. Constitutive Law Discovery (a)

  • For constitutive-law task (a), trainable Young’s modulus and Poisson’s ratio parameters are used to compute Kirchhoff stress from deformation gradients.The implementation initializes the parameters from the best values from training curves.Source: §D.1 · p. 18

D.2. Constitutive Law Discovery (b)

  • For task (b), a trainable gamma parameter corrects deformation gradients toward the identity when their determinant deviates from 1.The correction uses tanh(J - 1) to scale the adjustment.Source: §D.2 · pp. 18–19

D.3. Constitutive Law Discovery (c)

  • For task (c), the implementation applies a polar decomposition, a hardening-adjusted elastic limit, and a volume-preserving correction.The elastic limit and hardening factor are trainable parameters.Source: §D.3 · p. 19

D.4. Constitutive Law Discovery (d)

  • For task (d), a trainable scalar modulates corrections toward nearly isochoric behavior using volumetric and deviatoric components.The implementation combines the volumetric part with a correction based on the deviatoric deformation.Source: §D.4 · p. 20

D.5. Molecule Design (e); D.6. Molecule Design (f)

  • The appendix lists Top-20 optimized solutions for molecular-design tasks (e) and (f).The supplied passages do not describe the molecules or report their measured properties.Source: §D.5 · p. 20, §D.6 · p. 21

D.7. Molecule Design (g); 7. C1COC2C3CC4(NC2C34)O1

  • The cited passages identify a Top-20 solution set for task (g) optimized by the method, with c1cc(sc1Cl)C(C(F)(F)F)N listed as solution 1.Source: §D.7 · p. 22

D.8. Molecule Design (h)

  • Task (h) is accompanied by its Top-20 optimized molecular solutions.No molecular properties are supplied in the passage.Source: §D.8 · p. 22

D.9. Imaginary Constitutive Law

  • The imaginary constitutive-law implementation uses trainable kappa and mu parameters for volume and shape corrections to the deformation gradient.The code describes kappa as a bulk-modulus correction factor and mu as a shear-modulus correction factor.Source: §D.9 · p. 23
Loading 2405.09783v1…