PaperlayerView exact revision on arXiv

Source-linked AI summary

Delegating Earth science model calibration to knowledge-guided LLM agents

Zhanliang Zhu, Ziwei Li, Liujun Zhu, Yuchen Liu, Junliang Jin, Jianyun Zhang

arXiv:2610.18378v1cs.AIcs.CE

TL;DR

Applying process-based Earth science models requires calibration choices that reflect each application, but those choices are often dispersed across sources and depend on expert judgement. This paper uses knowledge-guided agents to define fixed experiments that a harness checks, executes, and assesses. The approach produced executable workflows across six model families, with snow-model tests showing first-attempt success for knowledge-guided specifications and further tests examining consistency, reproducibility, and fixed versus agent-proposed search.

  • Problem

    Calibration depends on application-specific choices about parameter ranges, observation mappings, metrics, and acceptance criteria, while model-interface and evaluation knowledge is dispersed and may not prevent scientifically mismatched experiments.

    Source: §1 · p. 2

  • Method

    Knowledge-guided agents use model infrastructure and declarations to author a calibration contract and runner, which the harness checks before fixed numerical search and assessment.

    Source: §1 · p. 2, §3.1 · p. 9

  • Results

    Across six model families, formulations executed and assessment identified unresolved targets; in FSM2, all five knowledge-guided specifications executed first try versus none of five model-name-only specifications.

    Source: §3.2 · p. 12

  • Takeaways & Limitations

    Fixed formulations make scientific choices inspectable and comparable, while numerical search and saved-evaluation replay support repeatable calibration without agent calls during search.

    Source: §4 · p. 21

    Each model family was tested at one site, leaving transfer across hydroclimates untested; acceptance depends on convention-specific bands and fallback rules.

    Source: §4 · p. 21

Abstract

from arXiv · show

Applying process-based Earth science models to new sites and questions often requires expert calibration decisions. Reliable delegation to language-model agents must preserve scientific competence and reproducibility. Here we develop a general calibration harness (GCH) for end-to-end delegation from a calibration request to assessed results. Constrained by a knowledge infrastructure (KI) and a validation convention, agents define parameter domains, observation mappings and objectives, and generate model-specific code. GCH checks the formulations against the same declarations and freezes them before numerical search and assessment, which require no further agent judgement. Agent-authored workflows executed across six model families, with assessment identifying unmet criteria. In snow-model tests, five of five knowledge-guided specifications executed on the first attempt, compared with none of five given only the model name, achieving validation Nash-Sutcliffe efficiency (NSE) of 0.81-0.94 against 0.57 for defaults. Under a common objective, independent GR4J formulations yielded scores within 0.001 NSE; re-evaluating eleven saved parameter vectors reproduced their metrics. In ten paired VIC tests with matched budgets, fixed search was non-inferior to continuing agent proposals (margin, -0.02 NSE; median difference, -0.003). This separation enables end-to-end calibration delegation with inspectable scientific choices and reproducible numerical evaluation.

1 Introduction

Earth-model applications require calibration choices tailored to their data, scientific aims, and evidence standards. This paper tests whether model knowledge can guide agents to formulate experiments that software can check, execute, and assess consistently.

1 Introduction

  • Each application requires matching model configuration to its data, scientific aim, spatial scale, and targets, then organizing calibration and evaluation for that application.The same catchment model can be judged against runoff alone or against runoff, snow storage, and evaporation together.Source: §1 · p. 1
  • Parameter ranges, observation mappings, metrics, and acceptance criteria encode prior scientific judgements that determine what optimization tests.These decisions are documented unevenly, making their reconstruction a substantial part of calibration effort.Source: §1 · p. 2
  • Language models have been tested as parameter proposers and workflow orchestrators, but formulation of the calibration experiment itself remains the role examined here [14] [15] [18].Prior work also considered executable environments and model knowledge packages [17] [18].Source: §1 · p. 2
  • The paper asks whether supplied model-interface and evaluation knowledge can guide experiment generation and let the same software check, execute, and assess that experiment.Both interface errors and scientifically mismatched choices can yield programs that run successfully while representing an incorrect experiment.Source: §1 · pp. 2–3

2 Methods

The methods package model-operating knowledge and calibration declarations into a workflow in which agents prepare a fixed experiment and deterministic software checks, searches, and assesses it. Experiments span six model families and include information-condition tests, repeated formulations, paired search, reference comparisons, and replay analyses.

2.1 Knowledge infrastructure and the general calibration harness

  • A model graph and validation convention encode permitted observation–metric combinations and sourced acceptance bands; the harness compiles objectives from these declarations and checks runners and parameter application before search.The graph is synthesized from five sources, while validation bands are sourced, verified, and frozen; the harness rejects undeclared outputs or metrics and treats failed or non-finite evaluations as infeasible.Source: §2.1 · pp. 3–4

2.2 Calibration contracts and runners

  • Agents author a machine-readable contract declaring targets, parameter handling, constraints, objectives, search strategy, budget, and holdout, plus a runner that applies parameters, executes the model, and reports metrics.Preparation includes generation, review or commissioning, and revision; each formal search then uses the fixed formulation.Source: §2.2 · p. 4

2.3 Assessment and component acceptance

  • Each observed variable is assessed against metric-specific losses, acceptance ceilings or fallbacks, correlation and baseline tests; a workflow passes only when every component passes.For the six-family application, validation data informed final-member selection, whereas FSM2 and paired VIC selected parameters on calibration data alone.Source: §2.3 · pp. 4–5

2.4 Six-family applications

  • Six families were tested across distinct observation structures, and their applications specified separate parameter sets, injection routes, observation mappings, objectives, and budgets.Cases included crops, snow-dominated and other catchments, a land-surface model, and an aquifer model; MODFLOW 6 used streamflow and filtered baseflow targets because head observations were unavailable.Source: §2.4 · pp. 5–6

2.5 Knowledge experiments with FSM2 and GR4J

  • FSM2 specifications were generated under structured-knowledge, raw-documentation, or model-name-only conditions; GR4J parameter ranges were separately compared with developers’ empirical ranges.FSM2 used five sessions per condition, a common task and DDS, while GR4J range overlap was assessed without calibration.Source: §2.5 · pp. 6–7

2.6 Repeated formulations and paired search

  • Repeated GR4J formulations were evaluated with shared runners and varied optimizer seeds, while paired VIC blocks compared fixed DDS search with continuing agent proposals at matched evaluation budgets.The VIC estimand was the median paired difference in validation NSE, fixed minus continuing, against a non-inferiority margin of −0.02.Source: §2.6 · p. 7

2.7 Reference comparisons

  • Reference comparisons evaluated generated GR4J and SAC-SMA workflows against model-specific references and parameter files.GR4J used a 1,000-evaluation DDS search versus 234 evaluations for the airGR reference; SAC-SMA used a declared 3,000-evaluation budget and reported calibration and validation metrics.Source: §2.7 · pp. 7–8

2.8 Numerical comparisons and replays

  • Numerical comparisons included SAC-SMA optimizer and budget variants, and eleven replay packages re-evaluated saved parameter vectors against archived records.The replay checks used the archived environment; the individual numerical results are listed in Tables B1–B24.Source: §2.8 · p. 8

3 Results

The section presents a knowledge-guided workflow in which agents formulate calibration experiments and a harness executes and assesses them. The applications span six model families with distinct observations and targets.

3.1 Knowledge-guided formulation across model families

  • A calibration task and model knowledge inform an agent-authored contract and runner, after which the harness searches the declared domain and assesses observed variables against declared criteria.Shared checks compile and commission formulations, enforce constraints and finite metrics, and return feedback during preparation before the formulation is fixed for formal search.Source: §3.1 · pp. 9–10, Figure 1 · p. 9
  • The six applications cover WOFOST, HBV, SUMMA, VIC, MODFLOW 6, and CRHM, each with its own observation structure and calibration formulation.Targets include crop measurements, catchment discharge, snow water equivalent, evapotranspiration, and filtered baseflow duration curves.Source: §3.1 · p. 10
  • Table 1 organizes the experiments by agent task, replication unit, and assessment, with paired periods denoting calibration and validation.Appendices D–G give protocols and retained records, while Table B2 lists outcomes and supporting material.Source: Table 1 · p. 10

3.2 Model knowledge resolves interface errors

Knowledge-guided snow-model specifications ran more reliably than model-name-only specifications, while multivariable assessment and fixed-formulation tests evaluated component acceptance, search consistency, and reproducibility.

3.2 Model knowledge resolves interface errors

  • All five knowledge-guided FSM2 specifications ran on the first attempt, versus none of five model-name-only specifications; validation NSE was 0.81–0.94 against 0.57 for defaults.Three of five raw-documentation specifications ran on the first attempt.Source: §3.2 · p. 12, Figure 2 · p. 13
  • Correcting observation mappings and timescale units made all five raw-documentation and three eligible model-name-only specifications runnable, raising validation NSE to 0.96–0.98 and 0.93–0.99, respectively.The repairs changed only those two decisions; two model-name-only specifications requesting nonexistent parameter bstb were ineligible.Source: §3.2 · p. 12, Figure 2 · p. 13

3.3 Multivariable assessment identifies unresolved targets

  • 16 of 23 variable–metric components met the joint acceptance rule across six models, and MODFLOW 6 was the only workflow to satisfy every component.Accepted counts were 0/2 for WOFOST, 4/5 for HBV, 3/4 for SUMMA, 2/4 for VIC, 4/4 for MODFLOW 6 and 3/4 for CRHM.Source: §3.3 · p. 14, Figure 4 · p. 17
  • MODFLOW 6 improved both duration curves, while assessment located unresolved crop-production and evapotranspiration discrepancies in WOFOST and VIC.VIC validation NSE for evapotranspiration fell from 0.05 to −0.29; WOFOST validation grain-mass error rose to 57%, above the 49% default.Source: §3.3 · pp. 16, 18, Figure 4 · p. 17

3.4 Consistent numerical evaluation of fixed formulations

  • Under a shared NSE objective, five independent GR4J formulations had a total NSE range of 9.2 × 10^-4, while replacing KGE with NSE shifted mean NSE by about 0.06 in two formulations.The largest within-formulation seed range was 2.3 × 10^-4.Source: §3.4 · p. 18, Figure 5 · p. 19
  • In ten paired VIC blocks, fixed search met non-inferiority to continuing agent proposals at a −0.02 NSE margin; the median paired difference was −0.003.The 95% bootstrap interval was −0.007 to 0.002, with 150 calibration evaluations per arm.Source: §3.4 · p. 18, Figure 5 · p. 19
  • All eleven replay packages reproduced their saved comparison metrics, but each replay covered one evaluation rather than a full seeded search or fresh agent generation.Three metric files were byte-identical.Source: §3.4 · pp. 18–19
  • At Ibex, validation NSE was 0.53 for SCE-UA, 0.50 for DDS and 0.43 for random search; Blanco River SAC-SMA reached NSE 0.70 versus 0.63 for the CAMELS benchmark.At Blanco River, KGE was 0.69 versus 0.70 for the benchmark, under a different forcing, snow and evapotranspiration configuration.Source: §3.4 · pp. 19–20

4 Discussion and conclusion

The discussion frames calibration as an explicit, auditable experiment definition whose choices determine what a model is tested against. It argues that freezing formulations before search supports comparison and reuse, while identifying unresolved targets and scope limits.

4 Discussion and conclusion

  • Calibration choices can be recorded in an executable contract and runner, including parameter bounds, observation mappings, metrics, and adequacy criteria.The VIC contract records choices such as excluding forcing corrections, dropping a parameter unsupported by generated inputs, and narrowing ranges to a feasibility constraint.Source: §4 · p. 20
  • Snow-model errors arose from interface names and time units despite numerically valid runs, while the SAC-SMA knowledge package omitted a bound that excluded a published reference configuration.The SAC-SMA reference used different forcing, snow, and evapotranspiration treatments, so the parameter sets represent different applications.Source: §4 · p. 20
  • VIC matched discharge while evapotranspiration diverged from its satellite product, and WOFOST matched calibration grain mass but not validation years while biomass stayed within its ceiling.These outcomes remain unresolved under the tested formulations; the record identifies the variable and criterion, while follow-up questions concern observation representativeness, parameter domains, objectives, and process representation.Source: §4 · pp. 20–21
  • Five independent GR4J formulations reached the same fit under a common objective, while changing the objective moved the result by two orders of magnitude more.Fixed-definition numerical search met its non-inferiority criterion without agent calls, and saved evaluations replayed exactly.Source: §4 · p. 21
  • Each model family was tested at one site, leaving transfer across hydroclimates untested; results also depend on recorded acceptance bands and the completeness of each knowledge package.The principal experiments used one language-model configuration, with three others tested for single-objective authoring.Source: §4 · p. 21

Data and code availability

  • The harness, knowledge infrastructure, models, optimizers, and observation sources are identified as available through linked repositories or archives.The Chinese discharge records for Xixian and Tangnaihai are subject to redistribution restrictions.Source: Data and code availability · pp. 21–22

Appendix A: Additional figures

The additional figures and tables document knowledge construction, calibration choices, and numerical-search comparisons across model applications. They also provide diagnostic and reference comparisons, while distinguishing experimental designs and assessment conventions.

Appendix A: Additional figures

  • Fig. A1 traces five source readers into a locked model graph and separately shows literature-informed validation bands checked and audited before freezing.The graph declares comparable observations and metric families; agents use it to formulate, while the harness uses it to compile objectives.Source: Figure A1 · p. 25
  • Fig. A2 compares range overlap with developers’ empirical 10th–90th percentile box across nine GR4J specifications; all model-name specifications selected Q rather than Qsim.The ranges were assessed without calibration.Source: Figure A2 · p. 26
  • Fig. A3 presents parameter-range, selection-rule, and assessment-criterion diagnostics; the supplied-range WOFOST screen had no finite losses and widened MODFLOW 6 validation remained inconclusive.The figure also projects a CRHM front onto calibration discharge and snow NSE, and compares selections for twelve targets across six families.Source: Figure A3 · p. 27
  • Fig. A5 compares generated GR4J and SAC-SMA calibrations with references, including fitted parameters and validation metrics; Fig. A6 compares optimizer and multiobjective search outcomes.Fig. A6 covers HYMOD search methods and budgets, NSGA-II against weighted DDS searches for HYMOD and WOFOST, and WOFOST backend fronts and component acceptance.Source: Figure A5 · p. 29, Figure A6 · p. 30

Appendix B: Additional tables

  • The appendix tables distinguish six-family formulation tests, multivariable direct-driver applications, and the FSM2 information ablation and GR4J range-and-target assessment designs.Table B8 reports declared-budget multivariable results, while Table B9 defines joint acceptance using finite losses, temporal correlation where applicable, and no deterioration from validation baseline.Source: Table B2 · p. 32, Appendix D · p. 32, Appendix E · p. 32

Appendix D; Tables B3–B5

  • Tables B3–B5 cover single-objective authoring, execution, cost, and blinded optimizer choices; every reported authoring produced a runnable specification.All selected DDS except one claude-fable-5 CRHM authoring, which selected SCE-UA.Source: Appendix D · p. 32

Appendix D; Tables B17 and B18

  • The passage lists 4 hand-written multivariable applications, 2 generated workflows, and 1 scalar-objective probe, concerning generated code and scientific choices and engine support for multivariable tasks.Source: Appendix D · p. 32

Appendix E; Fig. 2; Table B6

  • The GR4J range test compares knowledge-guided, documentation-only, and model-name specifications against developer parameter ranges and observation-target mapping.The static assessment was conducted without calibration.Source: Figure A2 · p. 26, Appendix E · p. 32

Appendix E; Fig. A5; Table B15

This appendix section reports reference calibrations, diagnostics of calibration choices, and tests of consistency under repeated formulations. It also identifies the numerical-reproducibility analysis as a distinct component.

Appendix E; Fig. A5; Table B15

  • The reference comparison includes one GR4J and one SAC-SMA calibration, with ten SAC-SMA reference seed files and reported fit, validation, and parameter comparisons.The Blanco River table contrasts standalone SAC-SMA with a coupled reference using different forcing and evapotranspiration setups.Source: Appendix E · p. 32
  • Assessment diagnostics cover ten parameter-range arms in five families, twelve cached target comparisons, and two selection rules on one CRHM front.These comparisons address constraints, scientific priorities, and assessment criteria.Source: Appendix E · p. 32
  • Five repeated GR4J formulations contribute three seeds each, with six objective interventions; the section reports consistency under a common objective and effects of objective choice.Numerical reproducibility is identified separately as another analysis.Source: Appendix E · p. 32

Appendix F; Table B11

The appendix organizes numerical-consistency and search diagnostics, including paired VIC searches, optimizer comparisons, and multiobjective evaluations. Its tables specify comparison designs and clarify that replaying saved evaluations is not the same as reproducing full searches.

VIC fixed search versus continuing proposals

  • The paired VIC comparison used ten blocks and 150 calibration evaluations per arm; it contrasts fixed DDS search after preparation with continuing agent proposals using validation-NSE differences.The table defines the difference as fixed minus agent search.Source: Appendix F · p. 32, Table B11 · p. 39

Consistency and numerical reproducibility

  • All eleven replayed parameter-vector evaluations reproduced their original numerical metrics, but full seeded searches and fresh agent generation were not repeated.Each replay used one specified vector with the archived environment.Source: Appendix F · p. 32, Table B12 · p. 39

Numerical diagnostics Optimizer and multiobjective comparisons

  • The appendix also includes 27 SAC-SMA optimizer runs, multiobjective and backend diagnostics, and paired definitions for numerical-search comparisons.The comparison figures cover hypervolume differences and retained Pareto-front sizes as well as component-acceptance decisions.Source: Appendix F · p. 32, Table B24 · p. 47

Formulation and authoring comparisons

  • The appendix reports six-family single-objective specifications and blinded optimizer choices; all authorings produced runnable workflows, while optimizer selections varied by authoring.The table of choices says all selected DDS except the claude-fable-5 CRHM authoring, which selected SCE-UA.Source: Table B3 · p. 33, Table B4 · p. 33, Table B5 · p. 34
  • FSM2 comparisons vary knowledge conditions, while the multivariable calibration table reports validation-informed final front-member selection and declared-budget results.The FSM2 table notes that constraints and caching affect the difference between proposals and completed executions.Source: Table B6 · p. 35, Table B8 · p. 36

Numerical diagnostics Optimizer and multiobjective comparisons

  • HYMOD optimizer tables report full-window NSE across six nominal budgets and three seeds, with optimizer repetition counters potentially exceeding nominal budgets.The excerpted rows include SCE-UA at budget 500 with best NSE 0.61 for seed 1 and 0.60 for seed 2, and DDS at budget 2,000 with best NSE 0.68 for seeds 0–2.Source: Table B19 · p. 43, Appendix F · p. 44

Implementation checks

The harness verifies agent specifications against declared model knowledge before numerical execution, including checks that the requested parameters are actually applied. In the FSM2 experiment, the pre-search check rejected every name-only specification and two raw-documentation specifications.

Implementation checks

  • Across five model families, commissioning, parameter-response and split-probe records confirmed output changes after parameter perturbations and distinct scored records for split-specific requests.Runner checks compare requested parameters with values read back from native model files; case-identity and input checks apply where specified.Source: Implementation checks · p. 48
  • In FSM2, the pre-search check rejected every name-only specification and two raw-documentation specifications before any model run.The model graph and validation convention make the environment verifiable by checking an agent's formulation against the description it used.Source: Implementation checks · p. 48

Calibration and validation data use

The section reports how the experiments used data, budgets and convergence checks across six model families. Sixteen of 23 assessed components met joint acceptance, and MODFLOW 6 met every component.

Calibration and validation data use

  • Validation data informed final-front selection in the six-family assessment, while FSM2 and paired VIC selected parameters on calibration data before validation.Repeated GR4J formulations, the GR4J reference comparison and HYMOD numerical comparisons scored the full optimization window without a separate validation period.Source: Calibration and validation data use · p. 48

Search budgets

  • Search budgets reflected active parameters and expected model cost; the paired VIC tests used 150 calibration evaluations per arm, while optimizer comparisons used fixed budget grids.Agents also drew on knowledge-infrastructure guidance and their own heuristics when declaring budgets.Source: Search budgets · p. 48
  • The current harness derives per-seed caps from pilot runtime, overhead and parallelism when given a wall-clock allowance, using three seeds by default.The pilot uses ten runs, or three when the user supplies a runtime estimate; the conservative runtime estimate applies a 1.2 multiplier to the 90th-percentile pilot runtime or the larger of the supplied estimate and slowest pilot run.Source: Search budgets · pp. 48–49

Convergence

  • Convergence is assessed separately from budgets and scientific acceptance, using optimizer-specific criteria and recording the termination reason separately from the convergence verdict.DDS has no native convergence test, so the harness reports a descriptive settle point while preserving its budget-dependent schedule.Source: Convergence · p. 49

Models, sites and observations

  • The six-family applications covered crop, catchment, land-surface, macroscale hydrology, groundwater and cold-region models at sites selected for two observed variables.CRHM was tested at Dore, where defaults already met criteria, and at Blue River, where the uncalibrated baseline was poor.Source: Models, sites and observations · p. 49

Agent-led formulation and numerical execution

  • 16 of 23 components met joint acceptance, and MODFLOW 6 met every component in the six-family assessment.Five flow cases met their selected numerical thresholds; the workflows used different discharge assessments for MODFLOW 6 and the other four cases.Source: Agent-led formulation and numerical… · p. 50

Workflow-generation examples

  • Generated WOFOST and CRHM workflows illustrate unit mapping, staged parameter activation and objective selection, while a CRHM baseline meeting its adequacy criterion stopped before optimization.The WOFOST example passed both soil-moisture components but failed the yield magnitude criterion; the CRHM probe removed timing families absent from model output.Source: Workflow-generation examples · p. 50

Direct-driver multivariable applications

  • All four direct-driver applications completed, but none met every component criterion; CRHM snow NSE reached 0.95 while discharge NSE fell from 0.71 to 0.23.VIC discharge NSE rose from 0.65 to 0.76, but validation evapotranspiration was unavailable because observations did not cover that period.Source: Direct-driver multivariable applications · pp. 50–51

Single-objective phase, authoring cost and cross-vendor authoring

The single-objective phase tested generated workflows, cross-vendor authoring and optimizer choices. All fifteen authorings across four configurations from two vendors produced runnable workflows, while retained records for the runs whose working directory was lost support the reported tables.

Single-objective phase

  • For poor-default-fit basins, DDS raised holdout NSE to 0.73 for HBV, 0.61 and 0.70 for two SUMMA seeds, and 0.71–0.73 for CRHM at Belly River.WOFOST, MODFLOW 6 and VIC met the holdout criterion after calibration; HBV, SUMMA and CRHM stopped before optimization when defaults already met their declared target.Source: Single-objective phase, authoring cost… · p. 51

Authoring cost and cross-vendor authoring

  • Fifteen of fifteen authorings across four configurations from two vendors produced runnable single-objective workflows; Kimi and DeepSeek were not tested due to unavailable headless authoring support.The initial claude-fable-5 authoring averaged 53,262 output tokens, US$10.05 and 23 minutes.Source: Single-objective phase, authoring cost… · p. 51

Optimizer-choice tests

  • Claude, GPT and Kimi agreed on all twelve optimizer choices across four framings, including choosing DDS over textbook SCE-UA for a rugged surface at moderate budget.The CRHM authoring that selected SCE-UA was the one within-class mismatch.Source: Single-objective phase, authoring cost… · p. 51

Record retention

  • The single-objective runs' working directory was cleared by a server reboot, but specifications and hashes were retained and numerical outputs survived in the coordinator session record.The reported runs therefore appear in no figure; Tables B3–B5 were transcribed from the console output.Source: Single-objective phase, authoring cost… · p. 51

Appendix E: Knowledge experiments and reference comparisons

The appendix compares knowledge conditions, repairs and reference calibrations. Knowledge-guided FSM2 specifications executed on first attempt more often than name-only specifications, while GR4J formulations conformed more closely to the supplied interface and empirical ranges.

FSM2 information conditions and calibration protocol

  • Five of five knowledge-guided FSM2 specifications executed without revision, versus none of five name-only specifications, and K validation NSE was 0.81–0.94 against 0.57 for compiled defaults.Three of five raw-documentation specifications executed without revision; the comparison used structured knowledge files that differed from raw documentation in content and volume as well as organization.Source: FSM2 information conditions and… · p. 52

Prespecified repairs and generation effort

  • After correcting the target name and converting hourly timescales to seconds, all five raw-documentation and three eligible name-only specifications reached validation NSE of 0.96–0.98 and 0.93–0.99, respectively.The unit correction improved validation NSE in all eight eligible pairs; two name-only cases containing out-of-interface bstb received no repair search.Source: Prespecified repairs and generation… · p. 52

GR4J parameter ranges and observation mapping

  • GR4J parameter-range overlap with the developers' empirical box was 0.81–1.00 for knowledge-guided, 0.41–0.53 for raw-documentation and 0.39–0.60 for name-only specifications.All nine specifications recovered X1–X4 and included the reference optimum; name-only specifications selected Q, which the runner did not expose.Source: GR4J parameter ranges and observation… · pp. 52–53

GR4J reference calibration

  • Both generated DDS and airGR reference calibration achieved NSE 0.80 and KGE 0.79 on the GR4J scoring window, using 1,000 and 234 evaluations, respectively.The reference hydrograph was unavailable, so the comparison is of scores and parameters.Source: GR4J reference calibration · p. 53

SAC-SMA fit and parameter references

  • The standalone SAC-SMA workflow achieved validation NSE 0.70, compared with 0.63 for the published CAMELS benchmark, but used different forcing and model structure.The authoring instructions supplied case, periods and reference-method information, so agreement with published choices is descriptive; none of the eleven fitted values fell within the reference range.Source: SAC-SMA fit and parameter references · p. 53

Appendix F: Consistency and numerical reproducibility

Repeated GR4J formulations produced tightly clustered NSE endpoints under a common objective, while matched-budget VIC tests found fixed search non-inferior to continuing agent proposals.

Repeated GR4J formulation

  • Five fresh GR4J specifications selected SCE-UA in four cases and DDS in one, with nominal budgets of 1,500–3,000 evaluations.SCE-UA stopped after 1,302–2,135 evaluations; two specifications that selected KGE were also searched with NSE substituted.Source: Repeated GR4J formulation · p. 53
  • Under NSE, the 15 GR4J endpoints spanned 9.22 × 10−4 and all rounded to 0.80; the largest within-specification seed range was 2.29 × 10−4.The 21 searches included five fresh specifications with three seeds each and six matched interventions; replacing KGE with NSE shifted mean NSE by about 0.06.Source: Repeated GR4J formulation · pp. 53–54

Fixed search and continuing agent proposals

  • The fixed-search versus continuing-agent median validation NSE difference was −2.99 × 10−3, satisfying the −0.02 non-inferiority margin.Median validation NSE was 0.87 for fixed search and 0.88 for continuing proposals; both arms used 150 calibration evaluations.Source: Fixed search and continuing agent… · p. 54

Reproduction of specified numerical evaluations

Saved-vector replays reproduced their comparison metrics, while optimizer, multivariable-search, range-selection, and sampling-adapter tests showed distinct performance and evaluation boundaries.

Reproduction of specified numerical evaluations

  • All eleven replay packages reproduced their comparison metrics, with repeated-GR4J, GR4J-reference, and FSM2 metric files byte-identical.The packages covered six family applications, the paired VIC experiment, repeated GR4J, two reference comparisons, and FSM2.Source: Reproduction of specified numerical… · p. 54

SAC-SMA optimizer comparison

  • At a nominal budget of 3,000, mean validation NSE was 0.53 for SCE-UA, 0.50 for DDS, and 0.43 for random search.These SAC-SMA comparisons used mean actual efforts of about 1,952, 3,000, and 3,000 evaluations, respectively.Source: SAC-SMA optimizer comparison · pp. 54–55

HYMOD and multivariable search comparisons

  • WOFOST’s NSGA-II, NSGA-III, and MOEA/D runs accepted one, two, and two of three components, respectively, so none met all component criteria.The HYMOD sweep compared DDS and SCE-UA over six nominal budgets; the separate CRHM knee was dominated at budget 60 but not 180, a descriptive diagnostic.Source: HYMOD and multivariable search… · p. 55

Parameter ranges and solution selection

  • CRHM discharge protection raised validation discharge NSE from −0.16 to 0.34 and snow NSE from −2.08 to 0.41, but neither selection was accepted.Accepted components rose from zero to two of four, while discharge performance remained below the initial baseline; neither WOFOST selection met joint validation acceptance.Source: Parameter ranges and solution selection · pp. 55–56

Sampling adapter

  • The HYMOD DREAM demonstration ran seven chains with 34 filled iterations and 119 post-burn-in states, but supplied no calibration endpoint.Convergence, effective sample size, likelihood adequacy, and uncertainty coverage were not evaluated.Source: Sampling adapter · p. 56

Related papers

Loading 2610.18378v1…