PaperlayerView exact revision on arXiv

Source-linked AI summary

When is a large language model a good calibration agent? Physics-labeled prompting for budgeted engineering simulator calibration

Ruo-Qian Wang

arXiv:2610.22379v1cs.LGcs.AI

TL;DR

Expensive simulator calibration depends on a limited sequence of evaluations, making early decisions important, while conventional optimizers may lack the physical information engineers use. This paper tests an LLM proposal policy conditioned on physical diagnostics and combines it with numerical continuation. The hybrid reaches an objective of 10^-4 in a median of 9 evaluations, compared with 17 for Nelder–Mead, while the physical-prompt advantage is strongest early and disappears by fifteen evaluations.

  • Problem

    Expensive simulator calibration is constrained by the cost of evaluations, while conventional numerical methods use parameter histories and can miss interpretable physical discrepancies that engineers use.

    Source: §1 · p. 3

  • Method

    The study tests an LLM that proposes bounded simulator parameters from evaluation history and physical diagnostics, then hands the accumulated evaluations to a numerical optimizer.

    Source: §1 · pp. 3, 5

  • Results

    At an objective of 10^-4, the continuation policy reaches target in a median of 9 evaluations, versus 17 for Nelder–Mead and more than 60 for standalone GP-BO and random search.

    Source: §8 · p. 17

  • Takeaways & Limitations

    The evidence supports using a physics-labeled LLM as a warm-start mechanism rather than as a superior standalone optimizer.

    Source: §6 · p. 16

    The LES calibration is effectively one-dimensional given its observables, so search-efficiency conclusions may differ for higher-dimensional closure calibration or observables that constrain C_k and C_e.

    Source: §7 · p. 17

Abstract

from arXiv · show

Calibrating expensive engineering simulators requires tuning model parameters to match reference data, a process heavily constrained by computational costs. This study investigates whether Large Language Models (LLMs), conditioned on physical diagnostic descriptions of simulation errors, can propose informative parameter updates more efficiently than conventional derivative-free optimizers. We test this on a cylinder-wake Large-Eddy Simulation and a damped-oscillator inverse problem. Physics-labeled prompts are evaluated against four text ablations (removed, anonymized, padded, misdirected) and standard optimizers (Nelder-Mead, Bayesian optimization, CMA-ES, and random search). Results demonstrate a strong early semantic advantage: the first physics-informed LLM proposal improves the initial design in 23 of 25 trials. After five evaluations, the median calibration error is 21.9 times lower than physics-free controls. However, this advantage is transient - disappearing by fifteen evaluations - and inherent LLM decoder variability yields a 9.8-fold performance spread. To address this, we introduce a hybrid workflow that uses the LLM as a warm-start, passing accumulated evaluations to a conventional optimizer. This two-stage numerical continuation policy reaches target accuracy in just 9 simulator runs, outperforming the LLM alone (12 runs), Nelder-Mead (17 runs), and Bayesian optimization (>60 runs). Supported by sensitivity analysis validating the test cases, this hybrid physics-labeled LLM scheme proves highly effective as a calibration agent under strict evaluation budgets.

Nomenclature

The nomenclature defines the bounded parameter space, calibration objectives, diagnostic inputs, and evaluation budgets used in the study.

Parameters and objectives

  • x denotes the bounded simulator parameter vector, while X is the feasible set defined by its bounds.J(x) is the calibration objective, with lower values indicating better calibration; JLES and Josc denote the LES and oscillator objectives.Source: Nomenclature · p. 3

Diagnostics and features

  • d(x) and q(x) denote the diagnostic vector and its text summary, and f_m and f* denote the simulated and target feature values.s_m and w_m are the normalizing scale and weight for feature m.Source: Nomenclature · p. 3

Budgets and other symbols

  • B counts simulator runs, B_switch is the LLM handoff budget set to 15, and τ is the objective threshold for budget-to-target comparisons.n denotes repeated runs of a policy; ν is molecular kinematic viscosity.Source: Nomenclature · p. 3

Model-specific terms

  • C_k and C_e are Smagorinsky closure coefficients, C_D and C_L,rms are mean drag and RMS lift coefficients, and k and c are oscillator stiffness and damping.CMA-ES, GP-BO, LES, LLM, NM, and OPRO abbreviate the named optimization strategies, simulation, model, and proposal method.Source: Nomenclature · p. 3

1. Introduction

The introduction frames simulator calibration as a costly sequence of decisions where early choices matter, and asks whether physical diagnostics can make LLM proposals useful under tight budgets. It presents a physics-labeled proposal policy with numerical continuation, reporting an early semantic advantage that fades and a hybrid policy that reaches target accuracy in fewer evaluations than standalone methods.

Motivation

  • Calibration costs are dominated by the outer loop, so a handful of early decisions can determine whether a usable result is reached before the simulation budget runs out.The number of evaluations determines computational cost, while early decision quality affects how efficiently that cost is incurred.Source: §1 · p. 3
  • Conventional optimizers infer search directions from numerical history, whereas engineers use interpretable discrepancies but bring labor-intensive, subjective, and difficult-to-reproduce judgment.Model-based optimizers need accumulated evaluations to learn parameter-objective relationships, leaving a gap in the early regime.Source: §1 · p. 3

Research question and approach

  • The study asks whether LLMs can turn physical discrepancies into specific interventions that improve simulator output, rather than merely answer questions about physics.Removing, anonymizing, length-matching, or misdirecting diagnostic language while holding numerical history fixed provides a controlled test of physical content.Source: §1 · pp. 3–4
  • The proposed policy combines optimization history with physically interpretable diagnostics to choose bounded simulator evaluations, then hands accumulated evaluations to a numerical optimizer.The contribution formulates this physics-labeled proposal policy and pairs it with continuation to Nelder–Mead or GP-BO at a fixed handoff budget.Source: Figure 1 · p. 4, §1 · p. 4

Contributions and findings

  • At an objective of 10^-4, the continuation policy reaches target in a median of 9 evaluations, versus 12 for the pure LLM policy and 17 for Nelder–Mead.Standalone GP-BO and random search require more than 60 evaluations; the sensitivity analysis found only molecular viscosity identifiable from the four force statistics.Source: §1 · p. 5
  • The two compared conditions have median best objectives 21.9 times lower after five evaluations and 4.7 times lower after ten, and are indistinguishable at fifteen evaluations.Source: §1 · p. 5
  • Five identical-policy arms have five-evaluation medians spanning a factor of 9.8, quantifying substantial run-to-run LLM proposal variability.The introduction presents this as a resolution floor for five-seed LLM-optimizer comparisons.Source: §1 · p. 5

2. Related work

Related work contrasts numerical calibration's generality with its tendency to discard interpretable physical information, and describes both the promise and evaluation challenges of LLM proposal mechanisms. The LES case is presented as a costly, diagnostic-rich test with weak constraints on closure coefficients.

2.1. Engineering simulator calibration

  • Derivative-free methods are general, but usually optimize anonymous coordinates and scalar objectives, discarding interpretable physical information engineers use to diagnose failed simulations.Budget-to-target statistics and data profiles emphasize evaluations to acceptable accuracy; weakly constrained parameters can make optimizer rankings uninformative.Source: §2.1 · p. 5

2.2. LLMs as proposal mechanisms

  • LLM optimization-by-prompting uses an evaluation history to propose candidates, but reported performance may depend on model size, prompt, task distribution, and actionable feedback.Repeated identical prompts can produce different proposals, so an LLM policy is a distribution over trajectories and single-arm curves confound policy quality with decoder sampling.Source: §2.2 · p. 5

2.3. LES calibration as an engineering test case

  • The cylinder-wake Smagorinsky LES case has costly evaluations and meaningful diagnostics, but its force-statistic observables weakly constrain closure coefficients.The paper reports sensitivity analysis alongside optimizer comparisons because this weak constraint affects interpretation of the benchmark.Source: §2.3 · pp. 5–6

3. Physics-labeled LLM calibration with numerical continuation

The method defines a bounded calibration proposal loop, then switches from physics-labeled LLM proposals to numerical continuation after a fixed budget. Prompt controls and separate comparison questions are designed to distinguish the effect of physical semantics from generic prompt structure and optimizer choice.

3.1. Calibration problem

  • The calibration problem evaluates bounded parameters through a simulator objective and can return a diagnostic vector plus a short text description of physical discrepancy.The prompt-based policy uses these inputs and returns a new bounded candidate.Source: §3.1 · p. 6

3.2. Numerical continuation

  • At the handoff budget, the two-stage policy transfers all evaluated parameter-objective pairs to Nelder–Mead or GP-BO and continues to the full budget.The switch is fixed at 15 evaluations, chosen a priori rather than tuned to results; early hybrid and standalone LLM runs share the same policy and initial design.Source: §3.2 · p. 6

3.3. Prompt controls

  • Prompt controls remove diagnostics, anonymize names, replace diagnostic language with length-matched filler, or deliberately misdirect physical labels.These imperfect controls retain numerical values and help distinguish semantic content from prompt length or number exposure.Source: §3.3 · p. 6

3.4. Comparison questions

  • The evaluation separates prompt ablations, standalone optimizer comparisons, and continuation comparisons so gains from a different optimizer are not mistaken for effects of physical prompting.The prompt comparison holds the LLM proposal mechanism fixed while changing semantic content; continuation compares the two-stage policy with its components.Source: §3.4 · p. 7

4. Experimental setup

The experiments evaluate budgeted calibration on cylinder-wake LES and damped-oscillator benchmarks, measuring best-so-far error, target attainment, and first-proposal success. Repeated trials and statistical comparisons characterize variability and differences between methods.

4.1. LES cylinder-wake calibration

  • The LES benchmark calibrates viscosity and two closure coefficients against four wake statistics from a higher-resolution run of the same solver, using physical interpretations of component discrepancies in the LLM prompt.The benchmark is framed as a self-consistent identifiability problem, not validation of two-dimensional LES against physical turbulence.Source: §4.1 · p. 7, Figure 2 · p. 8

4.2. Randomized damped oscillator

  • The oscillator benchmark samples hidden stiffness and damping targets within bounds, adds Gaussian observation noise, and tests 30 randomized trajectories per method at budgets of 15, 30, and 60 evaluations.Resampled targets also prevent the LLM from recalling a canonical parameter value.Source: §4.2 · p. 8

4.3. Performance measures

  • Performance is measured by median best-so-far objective at fixed budgets, simulator runs to reach fixed thresholds, and first-proposal improvements over the shared initialization.The target thresholds span inadequate calibration to the benchmark's empirical floor; runs-to-target directly translates into solver hours.Source: §4.3 · p. 8

4.4. Repeated runs

  • Because the LLM decoder varies across calls, repeated runs quantify policy variability; identical policies are pooled to give 25 repeats through 15 evaluations, while controls have 5–10 repeats each.The pooling permits prompt contrasts to be reported with uncertainty ranges rather than only median rankings.Source: §4.4 · pp. 8–9

4.5. Assessing the differences

  • The analysis uses medians and bootstrap 95% confidence intervals, permutation tests for arm differences, Fisher’s exact test for hit rates, and paired Wilcoxon tests on oscillator trajectories.Medians address strongly skewed LES objectives; the paired oscillator comparison uses the same randomized targets across methods.Source: §4.5 · p. 9

5. Results

Physics-labeled prompts improve early calibration, but their advantage fades as evaluations accumulate, while repeated identical prompts vary substantially. Passing the accumulated evaluations to an optimizer helps reach target accuracy with fewer simulator runs.

5.1.1. The first informed proposal

  • At evaluation 5, physics-labeled prompts improve on the best shared initialization in 23 of 25 runs (92%), versus 8 of 30 runs (27%) for ablated prompts.The same four initialization points were provided to every arm, making evaluation 5 the first diagnostic-influenced decision.Source: §5.1.1 · p. 9, Table 3 · p. 11

5.1.2. Size and persistence of the effect

  • The median best objective is 21.9 times lower than pooled controls at five evaluations and 4.7 times lower at ten, but the advantage is absent by fifteen.The length-matched control is the cleanest comparison for separating physical meaning from prompt length.Source: §5.1.2 · pp. 9–10, Table 2 · p. 10

5.2. Run-to-run variability of the proposal policy

  • At five evaluations, identical physics-labeled policies produce five-seed medians spanning a factor of 9.8, narrowing to 1.7× at ten and 1.2× at fifteen.The authors caution that five-seed comparisons at low budgets cannot resolve differences smaller than about an order of magnitude.Source: §5.2 · pp. 10–11, Figure 3 · p. 11, Table 4 · p. 12

5.3. Optimizer comparison and numerical continuation

  • After the B_switch=15 handoff, Nelder–Mead continuation reaches 7.32 × 10^-5 at B=60 and beats the pure LLM policy on all five seeds.GP-BO continuation reaches 7.32×10^-5, versus 1.24×10^-4 for standalone GP-BO.Source: §5.3 · p. 12, Figure 4 · p. 13

5.4. Budget to target

  • At τ=10^-4, GP-BO continuation reaches the threshold in 9 evaluations, versus 11 for Nelder–Mead continuation, 12 for the physics-labeled policy, 17 for Nelder–Mead, and >60 for standalone GP-BO and random search.Only the GP-BO continuation leads every alternative at every threshold; random search ties the continuations at τ=10^-3.Source: Table 6 · p. 13, §5.4 · pp. 13–14, Figure 5 · p. 14

5.5. Sensitivity and identifiability of the LES benchmark

  • The LES objective is governed mainly by ν; force statistics identify molecular viscosity but leave C_k and C_e weakly constrained.The authors caution that rankings after roughly fifteen evaluations reflect benchmark resolution, since very different parameter vectors lie within a 1.2% objective band.Source: §5.5 · pp. 14–15, Figure 6 · p. 14

5.6. Proposal hygiene and failure accounting

  • Nearly one in five standalone physics-labeled proposals duplicates an evaluated point, while continuation lowers duplicate rates to 0.029 and 0.007.The 30-evaluation batch also had 15 nondeterministic infrastructure failures charged to budget, 11 in the physics-labeled arm and four in the no-diagnostic arm.Source: §5.6 · p. 15

5.7. Oscillator replication

  • On the oscillator, the physics-labeled policy leads at B=15, but Nelder–Mead overtakes it at B=30 and wins on 27 of 30 trajectories at B=60.The randomized hidden targets reproduce the early-but-transient pattern; the misdirected prompt is not significantly worse on this smoother two-parameter task.Source: §5.7 · pp. 15–16, Table 8 · p. 16

6. Discussion

The evidence supports using a physics-labeled LLM as a warm start, not as a superior standalone optimizer: its early advantage fades, and continuation removes the stall. The interpretation is bounded by decoder variability and uncertainty about whether physical reasoning itself drives the early gain.

6. Discussion

  • A physics-labeled LLM is best treated as a warm-start mechanism: it leads early, then stalls, while numerical continuation removes the stall.The continuation policy is never worse than its components; the recommended handoff is after roughly ten to fifteen evaluations.Source: §6 · pp. 16–17
  • 19.6% of the standalone physics-labeled policy’s remaining budget is spent repeating evaluated points, helping explain its late stall.The continuation step largely removes duplicate proposals; at large budgets, the comparison is then between numerical optimizers.Source: §6 · p. 16
  • The early gain does not establish causal physical reasoning: the physical labels may supply a good first guess for the one identifiable parameter, viscosity.The authors treat physical explanations as an experimentally testable input channel, not evidence of causal reasoning about the simulator.Source: §6 · p. 16
  • A factor-of-9.8 spread in five-seed medians across identical-policy arms limits what low-budget comparisons can resolve.The paper recommends repeating policies and reporting policy repeats separately from seeds, then pooling them.Source: §6 · pp. 16–17
  • For costly, low-budget use, the paper recommends physically labeled diagnostics for the first few points, followed by a derivative-free optimizer and validation against sensitive observables.The recommended handoff is at a fixed budget of order ten to fifteen evaluations; the workflow also requires parsing, bound enforcement, duplicate handling, and failure policies.Source: §6 · p. 17

7. Limitations

The LES findings are constrained by limited seed coverage, weak identifiability of two closure parameters, and a two-dimensional testbed. Prompt controls, model configuration, and the fixed handoff budget also limit how broadly to generalize.

7. Limitations

  • LES results use five seeds; they are well-powered for the prompt contrast at B≤10 but under-powered for other comparisons.Policy repetition mitigates but does not eliminate under-sampling of seed-level effects.Source: §7 · p. 17
  • The nominally three-dimensional LES calibration is effectively one-dimensional for the chosen observables, so results may differ when closure parameters are constrained.Higher-dimensional closure calibration or observables sensitive to C_k and C_e remain natural next tests.Source: §7 · p. 17
  • The two-dimensional flow solver is a self-consistent inverse-problem testbed, not a validation of two-dimensional LES against physical turbulence.The calibrated coefficients therefore carry no physical interpretation.Source: §7 · p. 17
  • Prompt controls do not perfectly match all physical-prompt distributional properties, and the 30-evaluation batch has initialization failures.The authors recommend replicating that batch with infrastructure failures retried outside the optimization budget.Source: §7 · p. 17
  • Experiments use one LLM configuration and a handoff budget fixed at 15 without a sweep, limiting claims across models or handoff policies.Decoder variability should be re-measured when the model configuration changes; proposals also require validation and failure handling before safety-critical use.Source: §7 · p. 17

8. Conclusion

Physics-labeled LLM proposals improve calibration early, but their advantage is transient; handing accumulated evaluations to a numerical optimizer reaches the target in fewer runs than the compared alternatives.

8. Conclusion

  • 23 of 25 first physics-informed proposals improved the starting design; the median best objective was 21.9 times lower at five evaluations and 4.7 times lower at ten, but indistinguishable by fifteen.The comparison is against ablated prompts.Source: §8 · p. 17
  • A fixed-seed, identical-prompt policy still had five-evaluation medians spanning a factor of 9.8, setting a resolution floor for low-budget comparisons.The repeats show that decoder variability remains even when the seed fixes the initial design and numerical baselines.Source: §8 · p. 17
  • 9 evaluations reach the 10−4 objective threshold with the hybrid, versus 11–12 for pure LLM or permuted-semantics prompts, 17 for Nelder–Mead, and more than 60 for standalone GP-BO or random search.The hybrid hands the evaluated set to Nelder–Mead or GP-BO at a fixed budget.Source: §8 · p. 17
  • Only molecular viscosity is identifiable from the four force statistics, delimiting the budgets over which benchmark comparisons carry information.The paper recommends checking objective identifiability before ranking optimizers.Source: §8 · pp. 17–18
  • The resulting workflow uses physical semantics for the first expensive evaluations, then hands off to a conventional optimizer once numerical history is informative.The handoff addresses the late-budget stall of the LLM policy.Source: §8 · pp. 17–18

CRediT authorship contribution statement

The contribution statement assigns the listed research and writing roles to the First Author.

CRediT authorship contribution statement

Data and code availability

The experiment materials and analysis scripts are archived in Zenodo and supplied as supplementary material.

Data and code availability

  • The archive includes the harness, prompt templates, initial-design records, full per-evaluation logs for both LES batches and the oscillator benchmark, and scripts generating every table and figure.Run logs, rather than harness source, are authoritative for Appendix A; API credentials and cluster-specific paths are excluded.Source: Data and code availability · p. 18

Declaration of generative AI and AI-assisted technologies in the writing process

The appendix documents the prompt templates and objective details, while clarifying how control-template design and harness artifacts qualify interpretation of the experiments.

Declaration of generative AI and AI-assisted technologies in the writing process

  • AI-assisted writing tools helped organize and edit the manuscript; the author reviewed and edited it and takes responsibility for the final content.This is distinct from the LLM proposal policy evaluated as part of the experimental method.Source: Declaration of generative AI and… · p. 18

A. Prompt templates

  • Each prompt variant shares a preamble, case identifier, parameter bounds, best-trial history, incumbent diagnostic block, and closing instruction; templates are transcribed from run logs.Only the bounds and diagnostic blocks differ across variants, and the logs are authoritative because the released harness source was later edited.Source: A. Prompt templates · p. 18

A.1. Physics-labeled prompt

  • The physics-labeled prompt gives parameter bounds and physical roles, the best-trial history, incumbent feature diagnostics, and an instruction to return one bounded parameter vector as JSON.Its diagnostic examples identify errors such as low lift-shedding frequency and weak lift amplitude.Source: §A.1 · p. 19

A.2. Control diagnostic blocks

  • Controls replace the diagnostic block; anonymization renames parameters and removes units and role descriptions, while no-diagnostic, filler-padded, and misdirected variants provide other controls.The table notes that only the length-matched control matches the physics-labeled block; anonymized and no-diagnostic blocks are substantially shorter.Source: §A.2 · pp. 19–20, Table A.1 · p. 20

A.3. Remarks on the templates

  • At five evaluations, the length-matched control is 1303 characters versus 1299 for the physics-labeled block, but shorter anonymized and no-diagnostic contrasts confound label removal with text reduction.Source: §A.3 · p. 20
  • The permuted-semantics control is out of domain for the cylinder-wake parameters, yet exposes correctly named metrics, targets, and errors that may contribute to its competitive larger-budget behavior.The authors characterize it as a weaker negative control, not a directionally inverted mapping of the parameters.Source: §A.3 · pp. 20–21
  • Duplicated parameter-role text and wall-clock time passed as a metric were constant within the affected arms; the authors say neither invalidates comparisons but both should be removed in cleaner replication.Source: §A.3 · p. 21

B. Objective and feature extraction details

  • The LES objective uses force-coefficient time series over t∈[250, 350] to extract four features; the oscillator objective is mean squared displacement error on a fixed time grid.LES feature scales use target values for the first three features and π for phase, with all four weights set to unity.Source: B. Objective and feature extraction… · p. 21

C. Experiment configuration and reproducibility

Appendix C records the configuration and accounting choices behind the reported runs, while Appendix D details the statistical procedures used to analyze them.

C. Experiment configuration and reproducibility

  • Each case, seed, and method has one maximum-budget trajectory, with smaller budgets taken as prefixes; methods share the same four-point initial design.The common design supports paired and pooled comparisons.Source: C. Experiment configuration and… · p. 21
  • LLM proposals use gpt-oss:120b at temperature 0.1; mean inference latency is 4.2–8.1 s, compared with 267 s per simulator evaluation.Malformed responses are retried, parse success was 100%, and no proposal required projection.Source: C. Experiment configuration and… · p. 21
  • Continuation runs the physics-labeled policy to B_switch = 15, then gives the full evaluated set to Nelder–Mead or GP-BO for refinement.The analysis pools successful evaluations across batches and methods for sensitivity analysis because it characterizes the objective function.Source: C. Experiment configuration and… · pp. 21–22

D. Statistical procedures

  • The median and ratio of medians are used because right-skewed calibration errors make means sensitive to a minority of poorly performing runs.The reported contrast is the factor by which calibration errors differ, rather than their absolute change at a particular budget.Source: D. Statistical procedures · p. 22
  • Median intervals use 20,000 bootstrap resamples, while permutation tests compare ratios of medians without assuming an objective distribution.With n = 5 for anonymized and length-matched controls, intervals are necessarily wide.Source: D. Statistical procedures · p. 22
  • At B = 5, Holm-adjusted p-values of 0.0016, 0.0016, 0.040, and 0.040 leave all four individual comparisons significant.The pooled-control test is the primary comparison and is not affected by the four-test correction.Source: D. Statistical procedures · p. 22
  • First-proposal success counts use Fisher’s exact test; oscillator methods use paired Wilcoxon tests, and sensitivity correlations use Spearman ranks.The first-proposal contrast is 23 successes in 25 runs against 8 in 30 (p = 1.3 × 10^-6; odds ratio 31.6). Rank correlations indicate parameter information, while binned medians and the best-evaluation spread provide more direct evidence.Source: D. Statistical procedures · p. 22

Related papers

Loading 2610.22379v1…