PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Equation Search: Conjecture-Guided Discovery with SCOPE and ClosureBench

Yuxiao Hu, Qian Li, Ding Wang, Siheng Chen, Linfeng Zhang, Jinyue Yan, Yuntian Chen

arXiv:2610.18938v1cs.AIcs.LG

TL;DR

Equation fit alone does not determine whether a scientific conjecture should be retained, revised, or rejected, and existing benchmarks underassess the findings and evidence produced during discovery. SCOPE runs parallel conjecture-guided investigations, and ClosureBench evaluates interactive closure discovery. The paper reports improved equation recovery and generalization alongside more accurate, reliable, and effective scientific reports.

  • Problem

    Equation fit alone cannot establish whether an underlying conjecture should be retained, revised, or rejected, while existing benchmarks primarily assess equation recovery rather than findings and supporting evidence.

    Source: §1 · pp. 1–2, §2 · p. 3

  • Method

    SCOPE sustains parallel conjecture-guided scientist trajectories, and ClosureBench evaluates interactive discovery across 60 closure tasks in five domains embedded in known PDE systems.

    Source: §1 · p. 2, §4 · p. 6

  • Results

    SCOPE improves equation recovery and generalization on ClosureBench and real-world discovery tasks while producing more accurate, reliable, and effective scientific reports.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The work broadens automated discovery toward testing conjectures and assessing how reliably systems turn observations into scientific understanding.

    Source: §6 · p. 10

    ClosureBench currently covers local algebraic closures in simulated PDE systems, not differential, nonlocal, or history-dependent closures.

    Source: §6 · p. 9

Abstract

from arXiv · show

Most symbolic regression methods optimize candidate equations, but improving an equation's fit does not establish whether its underlying scientific conjecture should be retained, revised, or rejected. A conjecture may admit multiple mathematical realizations, and similarly accurate equations may imply different explanations. This motivates a shift from refining individual expressions to developing and testing the conjectures that guide them. We introduce SCOPE, a framework that sustains multiple conjecture-driven research trajectories. Multiple scientist agents independently pursue conjectures through equation modeling, question formulation, and active experimentation, then exchange evidence to revise their conjectures and research directions. Their findings are consolidated into a scientific report linking models, conclusions, and evidence. Existing equation-discovery benchmarks primarily assess recovered expressions, leaving the evidence and scientific conclusions established through the investigation underexamined. We therefore introduce ClosureBench, an interactive benchmark of 60 closure-discovery tasks across five scientific domains. ClosureBench focuses on closure-discovery tasks to give equation discovery a concrete scientific purpose. Each task embeds an unknown algebraic relation in a known PDE system, where agents control experimental conditions in OpenFOAM to acquire new data for equation discovery. Beyond equation accuracy, ClosureBench proposes three metrics to evaluate the discovery process: evidence faithfulness, mechanism insight, and experimental effectiveness. Compared with diverse baselines, SCOPE achieves higher equation accuracy on both ClosureBench and a real-world scientific discovery case, while producing more accurate, reliable, and effective scientific discoveries.

1 INTRODUCTION

Equation accuracy alone cannot determine whether a scientific conjecture should be retained, revised, or rejected, because conjectures can admit multiple equations and similarly accurate equations can support different explanations. SCOPE develops conjecture-guided investigations, while ClosureBench evaluates the evidence and findings produced through discovery.

1 INTRODUCTION

  • Most symbolic regression methods refine candidate equations, but fit alone cannot establish whether an underlying conjecture should be retained, revised, or rejected.A conjecture may have multiple mathematical realizations, and similarly accurate equations may imply different explanations.Source: §1 · pp. 1–2
  • SCOPE sustains parallel scientist-led research trajectories that connect conjectures, equation modeling, targeted experiments, evidence exchange, and evidence-grounded reporting.Scientists revise conjectures and research directions using new observations and roundtable discussions.Source: §1 · p. 2
  • SCOPE and ClosureBench improve equation recovery and generalization while producing more accurate, reliable, and effective scientific reports.The paper frames contributions around conjecture-driven research trajectories, a multi-scientist framework, and an interactive closure-discovery benchmark.Source: §1 · p. 2
  • ClosureBench introduces 60 closure-discovery tasks across five domains, embedding unknown algebraic relations in known PDE systems and evaluating discovery beyond equation accuracy.Its three additional metrics are Evidence Faithfulness, Mechanism Insight, and Experimental Effectiveness; agents acquire observations by setting OpenFOAM experimental conditions.Source: §1 · p. 2

2 RELATED WORK

Prior equation-discovery work includes symbolic regression, active experimentation, and interactive benchmarks, but evaluation has largely focused on recovering equations or structures. SCOPE organizes discovery around conjectures, and ClosureBench evaluates scientific findings and their evidential support.

2 RELATED WORK

  • Symbolic regression approaches include search-based, pretrained, and LLM-guided methods, while some newer systems incorporate active experimentation but remain organized around candidate equations.Examples include PySR, SR-Scientist, LLM-AutoSciLab, and LLM-ACES.Source: §2 · pp. 2–3
  • SCOPE organizes discovery around developing and testing scientific conjectures, with equations as one stage and reports linking models, evidence, findings, and uncertainties.Its overview describes parallel investigations using equation modeling, active experiments, and roundtable evidence exchange.Source: §2 · p. 3, Figure 1 · p. 3
  • ClosureBench embeds unknown algebraic closures in known PDE systems and uses OpenFOAM experiments to evaluate equation recovery, scientific findings, evidential support, and experimental progress.Earlier interactive benchmarks enable active exploration, but their primary criteria remain target-equation or structure recovery.Source: §2 · p. 3

3 SCOPE: SCIENTIFIC CONJECTURE-ORIENTED PARALLEL EXPLORATION

SCOPE maintains parallel scientist trajectories that share observations while retaining their own conjectures, equations, questions, and investigation records. It guides modeling and investigation with structured conjectures, targeted questions and experiments, and periodic cross-trajectory discussion.

3 SCOPE: SCIENTIFIC CONJECTURE-ORIENTED PARALLEL EXPLORATION

  • Each of SCOPE’s M scientists shares cumulative observations while maintaining a research state containing candidate equations, a conjecture, questions, and investigation records.The initial state also incorporates data analysis and literature findings.Source: §3 · pp. 3–4

3.1 CONJECTURE-GUIDED EQUATION MODELING

  • Structured conjectures combine mechanistic explanations, variables, qualitative relationships, predictions, candidate equation families, and evidence summaries to guide symbolic regression.Equation-subspace expansion refines, mutates, and recombines candidates connected to the conjecture; PySR or LLM-SR then optimizes equations, with elite candidates selected for low error, simplicity, and diverse variable dependencies.Source: §3.1 · p. 4

3.2 QUESTION-DRIVEN INVESTIGATION

  • Scientists use disagreements, residual patterns, and unsupported predictions to formulate questions, analyze evidence, and design feasible experiments with expected outcomes and falsification criteria.New observations are assessed against those criteria as supported, refuted, or inconclusive; scientists then revise conjectures and cite the evidence for revisions.Source: §3.2 · pp. 4–5

3.3 ROUNDTABLE DISCUSSION AND SCIENTIFIC REPORTING

  • Every two rounds, scientists compare trajectories and vote; without consensus, they test questions raised by other trajectories, while the final report grounds its claims in investigation evidence.After five roundtables, SCOPE selects the most-voted trajectory, breaking ties by lower mean prediction error; it returns that trajectory’s lowest-error final equation and retains alternatives.Source: §3.3 · pp. 5–6

4 CLOSUREBENCH

ClosureBench evaluates scientific discovery as a process of acquiring evidence and supporting conclusions, using hidden closure relations embedded in known PDE systems. Its tasks pair controllable experimental conditions with evaluation of equations, scientific properties, evidential support, and research progress.

4 CLOSUREBENCH

  • ClosureBench contains 60 closure-discovery tasks across five scientific domains and evaluates discovery processes as well as recovered equations.An OpenFOAM interface supports experimental evidence acquisition.Source: §4 · p. 6

4.1 CLOSURE-DISCOVERY TASKS

  • Each task embeds an unknown algebraic closure in a known PDE system, where the closure maps local physical inputs to a quantity in the governing equations.The known operator defines how the closure enters the system; controllable conditions include initial, boundary, or operating conditions.Source: §4.1 · p. 6
  • Researchers review generated closure candidates for dimensional consistency, physical constraints, and limiting behavior, then check their OpenFOAM implementations for numerical stability and conservation consistency.Tasks expose scientific context and experimental interfaces but keep the hidden closure undisclosed.Source: §4.1 · p. 6

4.2 CONDITION-DRIVEN INTERACTIVE EXPERIMENTS

  • Agents choose experimental conditions, and OpenFOAM produces observations from the resulting fields, requiring agents to seek conditions that address their scientific questions.Interactive methods may make at most 30 additional simulation calls and receive up to 3,000 observations; fixed-data and interactive methods share a maximum of 4,000 observations including the 1,000 initial observations.Source: §4.2 · pp. 6–7

4.3 EVALUATION PROTOCOL

  • Beyond equation metrics, Evidence Faithfulness checks whether report claims are supported by collected evidence, while Mechanism Insight scores claims against predefined properties of the hidden relation.The reference properties include variable dependence, trends, interactions, and limiting behavior.Source: §4.3 · p. 7
  • Experimental Effectiveness measures improvement in equation accuracy from initial to final equations on hidden test sets, and EF and MI judgments use a fixed, method-blind evaluator with human validation.Effectiveness is averaged across valid initial–final evaluation pairs and reported in percentage points; two domain-informed researchers independently annotate sampled reports for validation.Source: §4.3 · p. 7

5 EXPERIMENTS

Experiments compare equation accuracy and discovery quality across fixed-data and interactive methods. SCOPE generally outperforms the baselines, while additional analyses examine simulation progress and conjecture diversity.

5 EXPERIMENTS

  • The experiments assess final-equation accuracy and, for agents that conduct computational experiments, experimental decisions, evidence use, and mechanistic explanations.The baselines include fixed-data symbolic-regression methods and interactive agents; SCOPE’s interactive comparison focuses on parallel conjecture-guided investigation versus serial trajectories.Source: §5 · p. 7, §5.1 · p. 8

5.1 BASELINES AND IMPLEMENTATION DETAILS

  • Fixed-data comparisons include conventional and LLM-based symbolic-regression methods, while interactive comparisons include ReAct and Reflexion-Exp, which retain experience serially.Main experiments use three independent random seeds, with an LLM-call ceiling of 1,000 per task for methods using LLMs.Source: §5.1 · p. 8

5.2 MAIN RESULTS ON CLOSUREBENCH

  • SCOPE generally outperforms fixed-data and interactive baselines in final-equation accuracy across three backbones and scores higher than ReAct and Reflexion-Exp on EF, MI, and EE.Table 2 reports overall ClosureBench results; fixed-data methods are not evaluated on EF, MI, or EE.Source: §5.2 · p. 8, Table 2 · p. 9

5.3 EFFECTIVENESS OF MULTI-ROUND EXPERIMENTS

  • After 12 additional simulation calls, SCOPE reaches 33.33% ID Acc@1%, exceeding ReAct’s final 26.67% and Reflexion-Exp’s final 30.56%.SCOPE finishes at 41.67%; its gains are faster in early and middle stages and smaller near the end of the budget.Source: §5.3 · p. 8

5.4 EFFECT OF PARALLEL CONJECTURE EXPLORATION

  • The Figure 2 comparison matches SCOPE and Single-Conjecture on simulation, equation-search, and LLM-call budgets, while the ablation copies one primary conjecture across branches.This removes conjecture diversity while keeping the parallel branch count and initial observations the same.Source: §5.4 · p. 8, Figure 2 · p. 9
  • SCOPE’s ID Acc@1% lead over Single-Conjecture widens from 1.11 percentage points initially to 7.78 points at the final checkpoint, alongside different questions and evidence gaps across branches.SCOPE reaches 41.67% versus 33.89% for Single-Conjecture at the final checkpoint; the proposed explanation is that diverse questions guide later modeling and revision as observations are shared.Source: §5.4 · pp. 8–9, Figure 2 · p. 9

6 CONCLUSIONS AND LIMITATIONS

The conclusion frames equation recovery as one part of sustained conjecture development through modeling and experimentation. ClosureBench’s current scope is local algebraic closures in simulated PDE systems.

6 CONCLUSIONS AND LIMITATIONS

  • The paper presents equations as both quantitative outcomes and tools for developing and testing scientific conjectures.SCOPE supports this process through parallel research trajectories and periodic evidence exchange.Source: §6 · p. 9
  • ClosureBench currently covers local algebraic closures in simulated PDE systems; differential, nonlocal, and history-dependent closures are identified as future extensions requiring richer models and experimental evidence.Source: §6 · p. 9
  • SCOPE and ClosureBench are presented as a direction for automated discovery that addresses question formulation, testing explanations, and revising scientific understanding alongside predictive models.The paper also positions the benchmark as a way to assess whether autonomous systems turn observations into scientific understanding.Source: §6 · p. 10

THE USE OF LARGE LANGUAGE MODELS (LLMS)

LLMs have specified roles in SCOPE, benchmark construction, scientific reporting, and evaluation. The paper also distinguishes these research roles from auxiliary manuscript and illustration assistance.

THE USE OF LARGE LANGUAGE MODELS (LLMS)

  • Within SCOPE, LLMs support scientist agents’ conjecture development, equation modeling, question and experiment proposals, evidence interpretation, and roundtable discussions.Source: THE USE OF LARGE LANGUAGE MODELS (LLMS) · p. 10
  • LLMs generate and structurally modify algebraic closures for ClosureBench, whose candidates receive human review and numerical checks.Review considers dimensional consistency, physical constraints, and limiting behavior.Source: THE USE OF LARGE LANGUAGE MODELS (LLMS) · p. 10
  • An LLM synthesizes task, trajectory, equation, and evidence information into reports, while a fixed, method-blind LLM evaluates EF and MI claims and scores are computed programmatically.Two authors independently annotate EF claims and MI claim–fact pairs for human validation; equation-accuracy metrics and EE do not rely on LLM judgments.Source: THE USE OF LARGE LANGUAGE MODELS (LLMS) · p. 10
  • The paper says LLM contributions to research conception, experimental design, or manuscript preparation did not warrant authorship or contributor status.Their other auxiliary uses included generating some Figure 1 cartoon characters and polishing manuscript language.Source: THE USE OF LARGE LANGUAGE MODELS (LLMS) · p. 10

A ETHICS AND PRIVACY STATEMENT

B SENSITIVITY TO CANDIDATE-SET CAPACITIES

The sensitivity study tests whether changing candidate-generation and candidate-retention capacities improves SCOPE. The default configuration offers the best balance across the reported outcomes, while larger capacities do not help consistently.

B SENSITIVITY TO CANDIDATE-SET CAPACITIES

  • With M = 3 scientists, the study varies candidate-generation and candidate-retention capacities around the default (4, 12, 4) configuration.All configurations use the same tasks, data, experimental budgets, stopping criteria, and three seeds; Qwen is the representative backbone.Source: B SENSITIVITY TO CANDIDATE-SET… · pp. 13–14
  • Narrowing candidate generation reduces all three metrics, while widening it slightly improves OOD Acc but lowers ID Acc and MI below Default.A larger candidate pool therefore does not improve the three outcomes consistently.Source: B SENSITIVITY TO CANDIDATE-SET… · p. 14
  • Reducing retention from K = 4 to K = 2 produces the clearest losses in OOD accuracy and MI; increasing K to 8 also does not improve results.The authors suggest too few elite equations may limit discrimination among mechanisms, while additional equations may be redundant.Source: B SENSITIVITY TO CANDIDATE-SET… · p. 14
  • The Default (4, 12, 4) configuration provides the best balance across equation recovery, generalization, mechanism insight, and candidate-set size.Within the tested range, increasing capacity does not yield a consistent benefit.Source: B SENSITIVITY TO CANDIDATE-SET… · p. 14, Table 3 · p. 14

C REPRESENTATIVE TASKS IN CLOSUREBENCH

ClosureBench presents representative closure-discovery tasks in five scientific domains, embedding hidden relations in known PDE systems. Across tasks, agents recover closure mappings from local inputs and observations generated through OpenFOAM simulations.

C REPRESENTATIVE TASKS IN CLOSUREBENCH

  • The five representative domains are reactive transport, energy and thermal sciences, environmental mass transfer, materials processing and rheology, and particulate and porous-media flow.The target is the complete symbolic mapping from observable closure inputs to output; benchmark variants satisfy stated physical and numerical conditions over declared operating ranges.Source: C REPRESENTATIVE TASKS IN CLOSUREBENCH · p. 14, Table 4 · p. 15

C.1 TASK OVERVIEW AND COMMON EVALUATION PROTOCOL

  • Fit, ID, and OOD observations come from complete OpenFOAM simulations, with OOD cases including at least one control outside the public control box.Predictions are evaluated against held-out closure observations only after the equation is locked.Source: §C.1 · p. 15

C.2 LOCAL REACTION SOURCE IN A ONE-SIDED HEATED CHANNEL

  • In the heated-channel task, agents infer the local reaction source from beginning-of-step concentration and temperature, with the source reconstructed from the discrete concentration balance.The relation approaches first-order consumption at low concentration, saturates with concentration, and strengthens exponentially with temperature.Source: §C.2 · pp. 15–16

C.3 HETEROGENEOUS ELECTROTHERMAL HEATING

  • In heterogeneous electrothermal heating, the target Joule source is QJ = σE2, using the same positive conductivity that determines the potential field.Candidates use step-start temperature, electric-field magnitude, and composition; the source is reconstructed from the transient-conduction balance.Source: §C.3 · p. 16

C.4 COUNTER-CURRENT SPRAY-TOWER ABSORPTION

  • In the spray-tower task, absorption vanishes with either reactant, decreases for coarser droplets, and combines reagent saturation with a temperature-dependent factor.The published target is the mean source reconstructed independently from the gas-solute and liquid-reagent balances, retaining the physical multiplier Sref = 0.3 s−1.Source: §C.4 · pp. 16–17

C.5 FLOW-INDUCED CRYSTALLIZATION

  • In flow-induced crystallization, shear promotes crystallization within a temperature window that reaches zero at the glass-transition and melting endpoints.The crystallinity balance reconstructs the rate directly, and the temperature balance divided by 18 K supplies an independent check.Source: §C.5 · p. 17

C.6 MOBILE–IMMOBILE EXCHANGE IN A DUAL-POROSITY MEDIUM

  • In the dual-porosity task, the signed exchange closure transfers solute between mobile and immobile pores, with positive values moving it to immobile pores and negative values remobilizing it.Both balances independently reconstruct the exchange; candidates recover dimensionless g rather than the physical source.Source: §C.6 · pp. 17–18

D.1 COMPUTATIONAL RESOURCES AND SOFTWARE

  • The PDE solvers and interactive ClosureBench experiments use OpenFOAM v2512, while symbolic regression uses PySR 2.0.0a2.Experiments run on a cluster with 8 NVIDIA H800 GPUs and 8 NVIDIA H200 GPUs.Source: §D.1 · p. 18

D.2 BASELINE IMPLEMENTATIONS AND HYPERPARAMETERS

This appendix section describes baseline implementations and shared retrieval tools alongside SCOPE’s conjecture-centered workflow, from evidence analysis and experiment design through trajectory exchange and structured scientific reporting.

D.2 BASELINE IMPLEMENTATIONS AND HYPERPARAMETERS

  • E2E receives at most 200 observations and refines 10 candidate trees; DSR samples 100,000 candidates, and uDSR adds a genetic-programming population of 100 for 20 generations.DSR uses batch size 512, learning rate 5×10−4, and the listed arithmetic, trigonometric, exponential, and logarithmic operators; uDSR sets crossover and mutation probabilities to 0.5.Source: §D.2 · p. 18
  • PySR runs 15 populations of 33 for 125 iterations, while LaSR runs 10 populations of 33 for 25 iterations and retains at most 20 concepts.Both use PySR’s operator sets and nesting constraints; PySR caps expression size at 50, and LaSR at 30.Source: §D.2 · p. 19
  • LLM-SR uses 10 islands and 1,000 candidate samples, while SR-Scientist allows 25 tool turns per outer iteration and 40 outer iterations.LLM-SR uses NumPy+BFGS and at most 10 free parameters; SR-Scientist’s evaluator optimizes at most 10 parameters using BFGS.Source: §D.2 · p. 19
  • ReAct follows one reasoning–action trajectory without added planning or cross-round memory, while Reflexion-Exp adds persistent reflections after every three successful experiments.Both require an initial symbolic-regression tool call; Reflexion-Exp otherwise retains ReAct’s process and tool interfaces.Source: §D.2 · p. 19

D.3 SHARED LITERATURE-RETRIEVAL TOOL

  • All interactive agents share a retrieval interface and maximum budget, using deduplicated, relevance-ranked Semantic Scholar metadata and, when available, verified open-access passages.When full text is unavailable, the system retains abstract-level evidence and adds retrieved material with source information to the agent’s public evidence context.Source: §D.3 · p. 19

E DETAILED ALGORITHM OF SCOPE

  • SCOPE uses three scientists over 10 discovery rounds, with roundtables every two rounds; each scientist keeps its own conjecture, equations, questions, and research log while sharing observations.Initialization creates shared candidates and three distinct conjecture drafts, which scientists independently confirm and refine.Source: E DETAILED ALGORITHM OF SCOPE · p. 20, §F.1 · p. 21, §F.1.1 · p. 21

F.3 PROGRAM VERDICTS AND COMPLETE CONJECTURE REVISION

  • SCOPE prompts guide scientists to select consequential open questions, then revise complete conjectures using program-computed prediction verdicts while preserving unresolved questions and evidence limits.Prediction verdicts apply to registered predictions, not the whole conjecture; a not_obtained result is inconclusive rather than a refutation.Source: §F.2.1 · p. 23, §F.3 · pp. 25–26

F.2 QUESTION-DRIVEN ANALYSIS AND EXPERIMENT PROPOSAL

  • Scientists first assess existing evidence to identify what is resolved and what gap remains before proposing an experiment.The analysis compares residuals, input coverage, condition contrasts, and equation properties including dependencies, interactions, local behavior, parameter stability, and identifiability.Source: §F.2.2 · p. 24

F.2.2 ANALYZE EXISTING EVIDENCE BEFORE PROPOSING AN EXPERIMENT

  • Analysis of a selected question uses public evidence and available results to determine resolved parts, remaining gaps, candidate implications, and required observations.If evidence already resolves the question, the analysis should say so; otherwise it should specify the needed observation without treating a local state as an executable control choice.Source: §F.2.2 · p. 24

F.2.3 PROPOSE AN EXECUTABLE, DISCRIMINATING EXPERIMENT

  • When a consequential gap remains, propose one feasible experiment using controllable conditions, required measurements, registered outcome interpretations, and criteria that could challenge the conjecture.Predictions are registered before the experiment runs; if no experiment is warranted, identify the existing evidence that resolves the question.Source: §F.2.3 · pp. 24–25

F.4 ROUNDTABLE EXCHANGE AND INDEPENDENT VOTE

  • At each roundtable, scientists independently compare a shared snapshot and vote on trajectories without replacing conjectures; reports summarize consequential tests, evidence, and interpretive changes.The report follows the investigation and its evidential limits rather than reproducing deliberation or individual tool calls.Source: §F.4 · p. 26, §F.5 · p. 27

F.5 SEVEN-SECTION SCIENTIFIC REPORT WRITER

  • The seven-section report links the unknown closure and evidence scope to findings, numerical behavior, evidence boundaries, and recommended experiments while preserving unresolved alternatives.It should use only the supplied public digest, present finished scientific prose rather than an action chronology, and avoid internal IDs and checker or verdict labels.Source: §F.5 · pp. 27–28

G EVALUATION METRICS AND DESIGN RATIONALE

The section separates final-equation accuracy from the quality of the discovery process, since accurate fits alone do not establish mechanistic understanding or evidential support. It defines complementary metrics for equation recovery, experimental improvement, mechanism identification, and evidence faithfulness.

G.1 EQUATION-RECOVERY METRICS

  • NMSE and accuracy-to-tolerance evaluate each final equation separately on fixed in-domain and out-of-domain test sets.NMSE normalizes mean squared error by reference-output variance and uses all test points.Source: §G.1 · p. 29
  • Acc@1% and Acc@0.1% count a task as solved when the worst relative error among the best 95% of test points is within the specified tolerance.The reported task score is binary: 1 if solved and 0 otherwise.Source: §G.1 · p. 29

G.2 SCIENTIFIC DISCOVERY QUALITY

  • Symbolic Accuracy evaluates complete symbolic equivalence, but cannot distinguish no insight from an incomplete equation that captures part of a mechanism.ClosureBench instead uses Mechanism Insight to score report claims against frozen reference facts, including partial recovery.Source: §G.2 · pp. 29–30
  • Experimental Effectiveness measures net change in the fraction of test points within 1% relative error, while Mechanism Insight and Evidence Faithfulness assess report content and its evidential support.Experimental Effectiveness does not attribute gains to any single experiment; the metrics capture distinct outcomes rather than one aggregate score.Source: §G.2 · p. 30

G EVALUATION METRICS AND DESIGN RATIONALE

  • Final equation metrics assess absolute numerical quality, while experimental effectiveness, mechanism insight, and evidence faithfulness separately assess improvement, mechanistic properties, and support from the discovery trajectory.These dimensions are reported separately because strong performance on one does not guarantee strong performance on another.Source: G EVALUATION METRICS AND DESIGN… · pp. 28–29, §G.2 · p. 30

H HUMAN VALIDATION OF THE LLM EVALUATORS

In the wind-turbine wake case, the report’s robust conclusions are a two-lobed structure and downstream decay, while the available evidence does not establish a unique representation or generative mechanism. The section also describes the case setup, the experiments and revisions, and human validation of the LLM evaluators.

H HUMAN VALIDATION OF THE LLM EVALUATORS

  • The fixed evaluator closely reproduces human judgments for evidence faithfulness and mechanism insight, with lower agreement on mechanism insight because it distinguishes complete from partial recovery.Most mechanism-insight disagreements are between adjacent recovery levels.Source: H HUMAN VALIDATION OF THE LLM EVALUATORS · p. 31, Table 6 · p. 31

I.1 MOTIVATION AND EXPERIMENTAL SETTING

  • The external case uses a published wind-turbine wake study as a reference, while SCOPE receives 1,000 observations and can make up to 30 batches of active queries across 10 rounds.Three scientists investigate coordinates normalized by rotor diameter; the reference expression and parameters are withheld from SCOPE.Source: §I.1 · p. 32

I.2 SCIENTIFIC REPORT

  • Initial observations show similar off-axis maxima, a central trough, and decreasing peak force downstream, but their coverage is uneven across transverse spans.The low-downstream observations sample only the positive lobe, while most high-downstream windows are one-sided.Source: §I.2 · p. 32
  • Experiments contradicted the centered single-peak conjecture, prompting revision toward a double-peaked shape.A later validation again observed the central trough, further weakening the centered Gaussian or super-Gaussian explanation.Source: §I.2 · p. 32
  • Matched centerline, peak-region, and tail comparisons support approximately stable peak locations and downstream amplitude decline, but do not establish constant width or strict spatial separability.Earlier high-downstream experiments omitted either the centerline or the peaks, motivating the matched comparison.Source: §I.2 · p. 32
  • Experiments targeting center curvature and the outer shoulder did not distinguish the shifted double Gaussian from the difference of Gaussians in the main observation region.The final review left both representations compatible with the evidence, and the experiment budget was exhausted without unanimity.Source: §I.2 · pp. 32–33
  • The robust wake conclusions are a two-lobed structure and downstream decay, not a unique representation or confirmed generative mechanism.Both the difference of Gaussians and shifted double Gaussian fit the principal observation region; the report cautions that low global error cannot distinguish them.Source: §I.2 · p. 33
  • The report identifies limits on inference: edge observations do not establish effective support or reliable extrapolation, and physical causes and cross-condition applicability require further experiments.Suggested follow-up tests target center curvature, edge behavior, blade-resolved loading, and changed inflow turbulence.Source: §I.2 · p. 33

I.3 EVALUATION AGAINST THE PUBLISHED STUDY

The published-study comparison finds that the locked candidate differs algebraically from the reference shifted double Gaussian while retaining low in-domain and out-of-domain NMSE. In a separate crystallization-report comparison, SCOPE gives the most complete account of the reference mechanisms among the three displayed reports.

I.3 EVALUATION AGAINST THE PUBLISHED STUDY

  • The locked candidate differs algebraically from the published shifted double Gaussian, with ID NMSE 2.1086 × 10−4 and OOD NMSE 4.1932 × 10−4.The reference model is specified by the published equation and parameters, while the candidate does not recover its strict expression.Source: §I.3 · p. 33
  • The report has 11 deduplicated mechanism claims against 10 reference facts: seven match completely, three partially, and one additional claim is unmatched but not contradictory.The maximum one-to-one matching weight is W = 8.5.Source: §I.3 · p. 33

J.1 COMPARISON OF THE THREE REPORTS

  • Among the three displayed crystallization reports, SCOPE gives the most complete account of the ground-truth mechanisms, including spatial confounding and the bounded temperature window.The comparison marks complete, partial, and contradictory mechanism matches with green, yellow, and red backgrounds, respectively.Source: §J.1 · p. 35

J.2 COMPLETE SCIENTIFIC REPORTS

The reports recover crystallization-rate closures from simulation evidence, identifying shear promotion and non-monotone temperature behavior while leaving uncertainty about the specific algebraic forms. Their findings also expose evidence gaps at domain boundaries and in conditions that could distinguish competing candidates.

Observed phenomenology

  • The temperature window is bounded by the glass-transition and melting endpoints, while shear promotion remains monotone over the sampled range.The reports describe temperature responses that rise to an internal maximum and then decline; one reports that the window’s turning point is comparatively well constrained.Source: §J.2 · pp. 35, 37

Scientific findings and evidence boundaries

  • The reports support a non-monotone temperature response and shear promotion, but their selected closures remain approximations that cannot be uniquely identified from the available observations.One report finds a locked form indistinguishable from a runner-up at less than 1% relative-error difference; another reports prediction correlation 0.9995 and a 1.1% relative-error difference between forms.Source: §J.2 · pp. 36–38

Recovered closure and scientific interpretation

  • The selected closure represents shear promotion and a temperature window, but its terms do not separately identify microscopic processes and do not preserve all process properties exactly.In one candidate, shear and crystallinity are inseparable, so the relative rate increase for a crystallinity increment changes with shear level.Source: §J.2 · pp. 36–37

Evidence boundary and numerical evaluation

  • In a further report, the locked form and alternative remain indistinguishable on the available record, and fitted constants a and b vary with the initial crystallinity seed.The analysis identifies conditions that could amplify prediction differences, but those observations have not been obtained.Source: §J.2 · p. 36

Observed phenomenology

  • A crystallization-rate peak along crystallinity reflects early nucleation enhancement competing with later depletion of unconverted material.One report places the initial seed on the rising branch; another describes a transition from pre-peak enhancement to post-peak weakening.Source: §J.2 · pp. 36–37

Evidence boundary and numerical evaluation

  • The locked form in one report has public NMSE 0.000532 at complexity 24, while the sampled temperature range of 347.7–424.9 K reaches neither the 320 K nor 440 K boundary.Three conditions predicted to distinguish the locked and band-leading forms were not acquired; another planned condition duplicated an existing one.Source: §J.2 · p. 37

Recommended discriminating experiments

  • Recommended experiments target conditions that distinguish candidate forms, test uncovered temperature and shear regions, and resolve responses near the crystallinity upper range.Recommendations also include extending wall-temperature sampling toward 320 K and increasing resolution on both sides of the internal temperature maximum.Source: §J.2 · pp. 36–38

Evidence boundary and numerical evaluation

  • Another locked expression has public NMSE 7.16 × 10^-7 at complexity 32, but the runner-up remains indistinguishable and the sine crystallinity factor is untested at higher crystallinity.The selected form belongs to a 2-form band among 701 candidates.Source: §J.2 · p. 38
Loading 2610.18938v1…