PaperlayerView exact revision on arXiv

Source-linked AI summary

SciExam for ENSO: Can AI Agents Build Climate Models?

Yinling Zhang, Langchen Liu, Dongbin Xiu, Xueyan Zou, Xu Kuang, Mengdi Wang, Shilong Liu

arXiv:2610.10513v1cs.AIcs.LGphysics.ao-ph

TL;DR

Open-ended scientific research lacks a known answer that can establish whether a new model is valid. SciExam for ENSO has agents build stochastic models from observations and evaluates them with hidden scientific tests; six of twelve systems outperform the published reference, and stronger models align with debated explanations of ENSO asymmetry.

  • Problem

    Open-ended research lacks a single correct answer for grading, while scientific models must be judged for validity beyond agreement with a target.

    Source: §1 · p. 2

  • Method

    SciExam has agents process ENSO observations, freeze self-written diagnostics, and build models that hidden graders test on statistics, variable dependence, and held-out forecasts.

    Source: §1 · p. 2

  • Results

    Six of twelve agent systems score higher than the reference, mainly through better reconstruction and forecasting, while the reference best reproduces ENSO statistics.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The stronger models reduce to forms compatible with either of two debated explanations of ENSO asymmetry, offering candidate mechanisms rather than only data fits.

    Source: §1 · p. 3, §6 · p. 11

    Two viable reductions score within 0.3 percentage points of each other (0.398 and 0.395), so the evaluation cannot distinguish which explanation their dynamics require.

    Source: §5.1 · p. 10

Abstract

from arXiv · show

Language-model agents are increasingly asked to carry out open-ended scientific research, yet their results are usually graded against a known answer, a rubric, or a language-model reviewer, none of which can tell whether a new scientific model is valid. The AI Science Exam for El Nino-Southern Oscillation (SciExam for ENSO) is a benchmark in which agents build low-order stochastic models of ENSO, the dominant mode of interannual climate variability, from real observations. Within a six-hour budget, agents process the observations, write their own diagnostics, which are then frozen, and develop a model using only these diagnostics as feedback. Hidden graders then test whether the model reproduces ENSO's statistics, recovers unobserved variables, and forecasts held-out years, and score a published model in the same way. Across twelve agent systems, six produce models that score higher than the published model, mainly through better reconstruction and forecasting. The simplified forms of the stronger models are each compatible with one of the two competing explanations of ENSO's warm-cold asymmetry, an open debate that the task never mentions. Controlled runs of the top system under varied information suggest that its scores do not come from recalling the dated observational record and that the information it receives shapes how it builds its model. SciExam for ENSO can thus evaluate agent research where no answer is known, and the results suggest that agents can already build competitive models whose structures bear on questions that scientists still debate.

1. Introduction

SciExam for ENSO addresses the challenge of evaluating open-ended scientific research without a known correct answer by having agents build models judged on independent scientific tests. Six of twelve systems beat the published reference, while their model structures also connect to debated explanations of ENSO asymmetry.

1. Introduction

  • Existing evaluations use known answers, rubrics, or language-model reviewers, but scientific research also requires results to be scientifically valid against existing work.Low-order dynamical models provide a testbed because competing hypotheses can be written as equations, compared, and tested against observations.Source: §1 · p. 2
  • SciExam addresses the lack of a known answer in open-ended research with hidden tests of model statistics, variable dependence, and forecasts, while agents create their own frozen diagnostics.The task separates observation processing, diagnostic development, and model development within six hours; grader scores and held-out data remain inaccessible.Source: §1 · p. 2
  • Six of twelve agent systems score higher than the reference, mainly through stronger reconstruction and forecasting, while the reference best reproduces ENSO statistics.Three additional systems come close to the reference.Source: §1 · pp. 2–3
  • Competitive agent-built models reduce to forms compatible with either of two debated explanations of ENSO’s warm–cold asymmetry, despite the task not mentioning them.The models offer candidate mechanisms, but the shared evaluation does not distinguish which explanation their dynamics require.Source: §1 · p. 3
  • Controlled runs suggest the top system’s scores do not depend on recalling calendar years, while literature and external data access alter its modeling choices and performance.With external data, it calibrates to an operational index rather than task observations and scores lower.Source: §1 · p. 3

2. Related Work

Prior long-horizon agent benchmarks and scientific-research benchmarks leave a gap: none tests an answer-free result using scientific criteria beyond agreement with a target.

2. Related Work

  • Existing scientific research benchmarks grade against known answers, while answer-free long-horizon benchmarks typically expose scores during the run; SciExam instead uses hidden scientific tests on a real physical system.Its tests assess statistics, coupling, and forecasts after the run.Source: §2 · p. 3

3. Benchmark

SciExam structures autonomous ENSO modeling as a staged research task, leaving scientific decisions to agents while fixing the required deliverables. Models are assessed on statistics, reconstruction, and held-out prediction, and twelve agent systems are evaluated under the same setup.

3. Benchmark

  • SciExam separates task materials from a reusable execution protocol, applying that structure here to agents building calibrated stochastic ENSO models from observations.The benchmark mirrors a research setting in which scientists receive observations, background material, and a modeling objective, then decide how to build and assess a model.Source: §3 · p. 3

3.1. Scientific Modeling Task

  • The modeling task is a low-order stochastic dynamical model representing variability in both the central and eastern tropical Pacific.Its five anomaly variables describe central and eastern Pacific sea-surface temperatures, western Pacific thermocline depth, central Pacific zonal current, and western Pacific zonal wind.Source: §3.1 · p. 4

3.2. Agent Workflow

  • Agents prepare data, develop diagnostics, then build and calibrate a model within a shared six-hour budget, with scientific decisions left to them.Their diagnostics are frozen before model development, providing the only feedback during that phase; hidden graders do not return scores.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3. Scientific Evaluation

  • Models receive 0–1 scores on statistics, reconstruction of withheld variables, and forecasts of held-out years, combined as a weighted sum and compared with a published reference.The periods are 1950–2019 for statistics, 1980–2014 for reconstruction, and 2015–2024 for prediction; persistence is the forecast baseline.Source: §3.3 · p. 5, Table 2 · p. 6

3.4. Experimental Setup

  • Twelve agent systems are evaluated, each pairing a language model with an agent scaffold, and systems are compared on their final checkpoints.The harness saves checkpoints every five minutes of eligible time; best checkpoints would have eight systems exceeding the reference rather than six.Source: §3.4 · p. 6

4. Results

Six agent systems surpassed the published reference on composite score, and most gained through stronger reconstruction and prediction despite the reference’s stronger statistical fit. Claude Fable 5.1 performs well across all three dimensions.

4. Results

  • The reference reproduced observed statistics best at 0.727, while most agent-built models scored higher on dynamical consistency and predictive skill than the reference’s 0.323 and 0.157.The reference was developed with observations from all three test periods, whereas agents received only 35 years of observations.Source: §4 · p. 6
  • Claude Fable 5.1 nearly matched the reference in statistics, recovered withheld variables in both directions, and beat persistence at all twelve forecast lead times.Its simulations reproduced observed event counts for all seven event types within the simulated 95% ranges.Source: §4 · pp. 6–7
  • Six of twelve agent systems exceeded the reference’s 0.378 composite score; Claude Fable 5.1 led at 0.515, while three more scored 0.345–0.369.Figure 3 ranks final scores and shows weighted contributions from statistical properties, dynamical consistency, and predictive skill.Source: §4 · p. 6, Figure 3 · p. 7

5. Analysis

The analysis examines which simplified structures sustain competitive models and how information access shapes the top system’s modeling choices.

5.1. Agent-Built Models Reenact the ENSO Mechanism Debate

  • Simplification raised Claude Fable 5.1’s score from 0.515 to 0.542 and GPT-6 Astra’s from 0.442 to 0.462; no analyzed model lost more than 0.014.The reductions were retained only when they preserved nearly all original scores and the temperature-distribution asymmetry; statistical scores often rose with a small cost in dynamical consistency.Source: §5.1 · p. 9
  • GPT-5.5’s two viable reductions scored 0.398 and 0.395, within 0.3 percentage points, so the evaluation cannot determine which mechanism its dynamics require.The same graders selected the reductions, and the literature’s mechanism question remains open.Source: §5.1 · pp. 9–10
  • Four of nine competitive models reduced to linear drift with multiplicative noise, and five to nonlinear drift with additive noise, matching the two debated explanations of ENSO asymmetry.The agents were not told of either explanation.Source: §5.1 · pp. 7, 9, Table 3 · p. 10

5.2. A Case Study of the Top System: Memorized and External Knowledge

  • Three additional six-hour runs changed calendar labels or enabled web access; each condition ran once, so comparisons cover one agent rather than repeated-run effects.The web conditions blocked the publication describing the reference model.Source: §5.2 · p. 10
  • Scored under each agent’s stated calendar convention, the anonymized run reached best and final scores of 0.482 and 0.463, versus 0.531 and 0.515 for the formal run, while reproducing ENSO’s main statistical features.A competitive model can therefore be built without knowing which years were observed.Source: §5.2 · pp. 10–11
  • Literature-only access yielded a 0.528 best score near the offline 0.531 but a different process-based sparse model; data-source access instead produced the lowest best score, 0.475.The literature-only model scored higher on statistics (0.798 versus 0.741) and lower on dynamical consistency (0.556 versus 0.632); both web runs ended at 0.386 and 0.406.Source: §5.2 · p. 11

6. Conclusion

SciExam for ENSO uses hidden, independent tests to evaluate agent-built stochastic models without a known answer. Six of twelve systems exceeded a published reference, and their simplified forms aligned with competing explanations of ENSO asymmetry.

6. Conclusion

  • SciExam for ENSO evaluates agents’ stochastic ENSO models from real observations with hidden, independent scientific tests requiring no known answer; six of twelve systems exceeded a published reference.The stronger models’ simplified forms were each compatible with one of the two debated explanations of ENSO asymmetry.Source: §6 · p. 11

A. Task Materials

The appendix details the materials, prompts, submission interface, and evaluation procedures used to run and score the agent-built ENSO models. It also reports how checkpoint selection affects the reference comparison and describes the run lifecycle and isolation controls.

A. Task Materials

  • Agents receive observations through 2014, a prescribed index-processing recipe, background on the recharge oscillator, and freedom to choose their model equations and mechanisms.Observations from 2015 onward are withheld; submissions implement the chosen equations, calibrated parameters, and a model card.Source: A. Task Materials · p. 14
  • The processing recipe defines five monthly anomaly indices from specified Pacific regions and variables, while the interface standardizes stochastic stepping and grader-run simulations, reconstructions, and forecasts.The model must pass a data check; unstable, erroneous, or non-finite submissions fail the validity gate.Source: A. Task Materials · pp. 14–15
  • The task proceeds through data processing, diagnostic writing, and modeling; diagnostics report discrepancies and become read-only before the agent develops its submission.Prompts encourage continued work and model changes when diagnostics identify a weak aspect or parameter tuning is exhausted.Source: A. Task Materials · p. 15

B.1. Evaluation Protocol

  • Twelve complete agent systems are compared, each combining a language model, scaffold, and service settings, with six hours of eligible working time per run.Each system is run once, and the harness saves distinct submission checkpoints throughout the run.Source: §B.1 · p. 16
  • The final checkpoint is the main comparison, while identifying the best checkpoint requires hidden-grader scoring of every saved copy after the run.Because the best checkpoint scores at least as high as the final one, reporting final checkpoints can only understate each run.Source: §B.1 · p. 16

B.2. Lifecycle and Isolation

  • Six systems exceed the reference by final checkpoints and eight by best checkpoints, with best checkpoints selected after runs using hidden graders.The main comparison uses final checkpoints, so it can understate each run's score.Source: §B.1 · p. 16, Table 4 · p. 17
  • Runs use isolated containers with graders, the reference model, and post-2014 observations kept outside; agents work offline and web tools are disabled.The lifecycle defines credited provider pauses, phase limits, and early-stop conditions.Source: §B.2 · p. 17

B.3. Scaffold Identities and Reproducibility

This section records scaffold and routing information, describes additional Claude Fable runs, and specifies the released materials for reproducibility.

B.3. Scaffold Identities and Reproducibility

  • Table 5 catalogs the scaffolds and routes used by the twelve systems; three additional Claude Fable 5.1 runs used its main configuration, with web access through a filtering proxy in two conditions.Context lengths reached 658,258 tokens for Claude Opus 5 and 314,562 tokens for Kimi K3.Source: §B.3 · p. 17, Table 5 · p. 18
  • The released code and data include the harness, graders, final models, checkpoint scores, and checksums identifying each component.Runs used the same task files, prompts, graders, and container image, and recorded their configurations.Source: §B.3 · p. 18

C. Evaluation Details

The graders score models on statistics, dynamical consistency, and predictive skill, using each submitted model in three distinct roles. These tests assess both whether simulations match observed ENSO behavior and whether models recover withheld variables and forecast unseen years.

C. Evaluation Details

  • The evaluation combines statistical, dynamical-consistency, and predictive-skill scores, applying the same submitted time-stepping model across all three tests.Models that do not remain finite are not scored, and each is evaluated using its agent-specified calendar convention.Source: C. Evaluation Details · p. 18

C.1. Statistical Properties

  • Long free-running simulations are compared with observations across variability distributions, seasonal variance, autocorrelation, event frequencies, and event strength-location patterns.The graders run 2,300 years, discard the first 200, and average statistics across thirty 70-year segments; a minimum across features prevents one failure from being offset by others.Source: §C.1 · pp. 19–20

C.2. Dynamical Consistency

  • Data assimilation tests whether observed variables let the model reconstruct withheld ones, providing evidence about whether its encoded relationships are realistic.An ensemble Kalman smoother assimilates monthly observations and revises the preceding three months; test A reconstructs hW, u, and τ from TC and TE, while test B reverses the roles.Source: §C.2 · pp. 20–21

C.3. Predictive Skill

  • Forecasts over the held-out period are compared with persistence, with the predictive score measuring improvement over that baseline.The graders initialize five model variables with observations and run a 50-member ensemble for 12 months without further observations; the held-out data include 2015–2024.Source: §C.3 · p. 21

D. Cost

The appendix reports run costs and presents Claude Fable 5.1 as an example of an agent-generated model and its simplified form. The simplified version retains linear coupling and state-dependent noise while trading lower dynamical consistency for improved statistical properties and predictive skill.

D. Cost

  • OpenRouter spending across all run batches was about $521; subscription systems paid fixed fees, so their token use was reported instead.The cost table covers model calls, prompt tokens, cache-served prompt-token share, and run costs; GPT-5.6-Sol usage was not recorded.Source: D. Cost · p. 22, Table 6 · p. 22

E.1. Claude Fable 5.1

  • Claude Fable 5.1 models five state variables with seasonally modulated linear interactions, nonlinear feedbacks, cubic damping, and state-dependent stochastic forcing.The appendix provides original parameters and auxiliary definitions, then describes the simplified model as retaining refitted linear coupling and the original state-dependent stochastic forcing.Source: §E.1 · pp. 23–24
  • The simplified Claude Fable 5.1 model removes nonlinear drift terms and confines seasonal drift variation to TC and TE growth and damping.It has 34 drift coefficients and improves statistical properties and predictive skill relative to the original, but lowers dynamical consistency.Source: §E.1 · p. 23

F. Claude Fable 5.1 under Four Conditions

The appendix compares Claude Fable 5.1 across four run conditions and describes how the study simplified agent-built models. Simplifications were retained only when scores and ENSO complexity remained sufficiently close to the originals.

F. Claude Fable 5.1 under Four Conditions

  • Table 11 covers four Claude Fable 5.1 conditions, each run once for six eligible hours, and reports best-checkpoint and final scores.Its component scores refer to the best checkpoint, and parentheses show changes relative to the offline run.Source: Table 11 · p. 26, F. Claude Fable 5.1 under Four… · p. 26

G. Structural Simplification of All Twelve Models

  • Model reductions were retained only if they preserved at least 95% of composite score, 90% of every dimension score, and the model’s ENSO complexity.When removing nonlinear drift and multiplicative noise nearly erased asymmetry in three models, reduction stopped; if both families of reductions qualified, the higher-scoring one was kept.Source: G. Structural Simplification of All… · p. 26
  • The three models scoring below 0.24 were screened on scores alone because they already missed several ENSO statistical features.Those models were not included in the mechanism analysis.Source: G. Structural Simplification of All… · p. 26

H. Further Related Work

The related-work discussion distinguishes SciExam for ENSO from equation-discovery pipelines and existing ENSO models. It positions the benchmark’s workflow and agent-built models as additions to research on ENSO asymmetry.

H. Further Related Work

  • Equation-discovery methods use sparse or symbolic regression, and language models can propose equation skeletons fitted and scored against data.These methods operate on prepared data within a fixed pipeline, unlike SciExam agents, which process raw observations and choose what to check.Source: H. Further Related Work · pp. 26–27
  • Prior ENSO models range from the recharge oscillator to stochastic diversity models, while an automated framework selects model terms using causal inference.That framework validated models through statistics, event diversity, and recovery of unobserved variables; SciExam adopts its first two criteria and adds held-out forecasts.Source: H. Further Related Work · p. 27
  • SciExam’s workflow has agents process raw observations, choose and freeze their diagnostics, and develop models without seeing scores; their models add candidates to the unresolved asymmetry debate.The competing explanations concern nonlinear dynamics and multiplicative noise.Source: H. Further Related Work · pp. 27–28
Loading 2610.10513v1…