Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental science

Maohao Ran; Chendong Ma; Yanting Zhang; Dailing Jiang; Yusen Huang; Meng Gao; Jun Song

Source-linked AI summary

Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental science

Maohao Ran, Chendong Ma, Yanting Zhang, Dailing Jiang, Yusen Huang, Meng Gao, Jun Song

arXiv:2608.18726v1Aug 19, 2026cs.CL

TL;DR

Environmental-science LLM evaluations often score only final answers, leaving quantitative execution unobserved. The paper introduces AtmosCoder-Bench, an execution-grounded benchmark with validated, uniquely verifiable problems and deterministic grading. It finds inflated multiple-choice accuracy, inconsistent application of known rules during multi-step computation, and persistent frontier-model weakness when task conditions invalidate familiar methods.

Problem
Existing environmental-science benchmarks provide limited systematic evidence about numerical-analysis ability because they generally leave the calculation process unobserved.
Method
AtmosCoder-Bench requires models to write and execute code for 436 certified, self-contained computational problems, exposing an auditable process and grading answers against deterministic ground truth.
Results
Across atmospheric and broader environmental tasks, multiple-choice formats inflate apparent ability by 12–39 percentage points, while models show inconsistent rule application and frontier models struggle to adapt methods to task-specific conditions.
Takeaways & Limitations
Execution grounding makes computational failures directly traceable and shows that domain expertise remains essential for adapting methods to unique physical conditions.
Takeaways & Limitations
The benchmark is verifiability-filtered, excluding open-ended or multiply answerable problems, and some reference answers rely on consensus among three LLMs.

Abstract

from arXiv · show

Large language models are increasingly used for quantitative work in the environmental sciences, yet existing evaluations score only final answers, leaving calculation process unobserved. Here we introduce AtmosCoder-Bench, an execution-grounded benchmark that makes the calculation process visible. Built through a transferable semi-automated pipeline (436 problems, 3,910 variants, 7,029 graded quantities), every problem is validated to be unambiguous and human-solvable, with uniquely verifiable answers. We find that (i) multiple-choice formats inflate measured accuracy by at least 12 percentage points; (ii) many failures arise not from missing knowledge but from models failing to apply known formulas and constraints consistently throughout multi-step computation; and (iii) even frontier models remain weak when task-specific conditions invalidate familiar methods, often reverting to canonical solution patterns rather than adapting methods to the relevant physical regime, leaving expert oversight essential.

1 Introduction

Environmental-science LLM benchmarks have largely measured final answers rather than observable computation, motivating an auditable execution-grounded evaluation. AtmosCoder-Bench exposes calculation processes and finds inflated multiple-choice performance, inconsistent multi-step rule application, and persistent failures to adapt methods to task-specific conditions.

Motivation: Environmental-science benchmarks cover knowledge, policy, and analysis, but systematic evaluation of numerical-analysis ability remains limited.Existing numerical benchmarks include AtmosSci-Bench’s symbolic expansion into multiple-choice questions and OneAtmos-Bench, while broader evaluation spans climate QA, policy assistance, and Earth-science analysis.
Motivation: Answer-only scoring obscures whether models actually execute the claimed calculations, while LLM judging and symbol matching introduce non-deterministic grading surfaces.Models may perform one calculation while claiming another, or fabricate intermediate steps without executing them.
Benchmark design: AtmosCoder-Bench pairs certified ground truth with required code execution, visible computation, and deterministic numerical grading across 436 atmospheric quantitative QA items.Its construction pipeline is extensible to broader environmental analysis and uses frontier-model validation plus human expert verification of reference answers.
Findings: 12–39 percentage points: multiple-choice evaluation overstates apparent LLM calculation ability, depending on model and whether defective answer keys are included.On identical problems, option-mode accuracy exceeds computed open-answer accuracy by 20–39 points, with a 12–27-point gap after defective keys are removed.
Findings: Models often know the correct formulas or rules but apply them inconsistently across long computations, constraints, and parameter choices.Execution grounding reduces truncation from natural-language reasoning and makes computational failures directly traceable.
Findings: Frontier models remain weak when task-specific conditions invalidate familiar methods, frequently reverting to canonical solution patterns instead of adapting to the physical or geographical regime.The benchmark identifies professional judgement to adapt methods to unique task conditions as a persistent gap.

2 Results

Execution-grounded evaluation exposes substantial variation in model calculation ability, including format-sensitive accuracy inflation, multi-step application failures, and persistent difficulty adapting methods to task-specific physical conditions.

Restrictive answer formats: 12–27 points of chance-corrected accuracy inflation remains after defective answer keys are removed, with option mode outperforming computed answers.Across the full analysis, the unadjusted gap is 30–45 points, and the remaining gap is robust to more permissive numerical tolerances.
Overall performance: 97.6% versus 41.1% separates the strongest and weakest execution-grounded configurations, while domain-adapted ClimateGPT models score near zero.The benchmark discriminates across current capability levels and includes an out-of-distribution reference.
Execution grounding: Execution grounding reduces truncation and natural-language calculation errors, although reasoning settings provide modest accuracy gains at substantially higher token cost.A difficult 11-sub-question problem was solved in fewer than 40 Python lines, whereas direct derivations can exceed output limits.
Robustness: Accuracy remains almost unchanged across numeric and paraphrase variants, preserving the leaderboard when the physics is fixed.These perturbations test whether performance reflects robust calculation rather than memorized wording or parameter patterns.
Failure modes: Models frequently know relevant formulas or rules but fail to apply them consistently through long chains, conservation bookkeeping, and iterative solutions.Observed errors include mixed units, lost symbols, broken constraints, truncated iteration, and compounded intermediate mistakes.
Domain variation: Performance is strongest for template-able, single-equation physics and weakest when tasks require regime judgement, sustained bookkeeping, or expert parameterization choices across environmental domains.The same capability ordering and discrimination appeared in hydrology, environmental chemistry, ecology and biogeochemistry, and soil mechanics.

3 Discussion

AtmosCoder-Bench uses rigorously verified tasks and execution-grounded evaluation to expose how LLMs calculate, revealing persistent failures in multi-step consistency and task-specific physical judgement.

3 Discussion: AtmosCoder-Bench rigorously removes ambiguity and requires uniquely verifiable, expert-solvable problems through semi-automated construction and multiple verification gates.The benchmark extracts textbook problems, validates answers, and reduces human labour while supporting scalable construction.
3 Discussion: Execution grounding requires models to write and execute Python solve() functions, making calculations visible and auditable at comparable accuracy and lower token cost than free-form calculation.Reasoning uses 1.3–2.1× as many tokens per full run under the execution-grounded architecture.
3 Discussion: 12–39 percentage points quantify the inflation of apparent calculation ability introduced by multiple-choice evaluation, depending on the model and defective-key inclusion.The reported range compares multiple-choice evaluation with computed open-answer performance.
3 Discussion: Models often know relevant formulas and rules but fail to apply them consistently across long computations, heterogeneous units, intermediate results, and multi-step constraints.Observed failures include truncation, lost signs, confused quantities, and unreliable constraint maintenance.
3 Discussion: Frontier models frequently reuse canonical methods when task-specific conditions invalidate them, revealing a calculation ceiling tied to professional judgement.In a Gaussian plume case, gpt-5.5 (reasoning) used Briggs parameters while claiming to use the required Turner coefficients.
3 Discussion: The benchmark cannot cover the full environmental domain because automated construction retains only independently verifiable, executable tasks and samples few additional fields.The authors report that many real quantitative problems have open-ended or multiple valid answers, and only 131 tasks span four fields beyond atmospheric science.

4 Methods

The methods construct a verified computational corpus, enforce auditable solver behavior, generate controlled variants and traps, and compare answer formats with replicated grading procedures.

4 Methods: 436 self-contained problems are extracted from 13 established textbooks across atmospheric and environmental sciences, then labelled by physical category and difficulty.The corpus includes 734 graded sub-answers spanning single-quantity and multi-quantity tasks.
4 Methods: Reference solve() functions are generated blind under a fixed Python contract requiring explicit inputs, unit conversions, and ordered keyed outputs.The solver-writing model does not see the target answer, and only the Python standard library is permitted.
4 Methods: Reference values are admitted through textbook matching or agreement among models from three different developers, with disagreements repaired, logged, or discarded.This process exposes misprinted keys and damaged extractions before values are frozen.
4 Methods: Deterministic audits reject answer literals and ignored inputs using abstract-syntax-tree inspection and input perturbation, followed by structural and consensus checks.The checks target grading collapse, question–answer misalignment, and unsolvable items.
4 Methods: De-scaffolding removes supplied formulas and recallable constants only after answer-recovery, data-integrity, and independent-reproduction checks.On 169 affected problems, removing scaffolding reduced the strongest model’s score by 1.8 points and a 9-billion-parameter model’s score by 18.5 points.
4 Methods: 346 of 436 problems receive five safe numeric variants each, preserving computational structure while recomputing ground truth at perturbed inputs.Physical constants are removed from solver signatures and outputs are verified bit-identical before perturbation.
4 Methods: LLMs act as proposers, while deterministic execution, textbook anchoring, cross-developer agreement, expert adjudication, and expert re-derivation determine admission.The physics content is taken from published textbooks rather than authored by models.
4.3 Trap diagnostic: 67 held-out traps minimally alter certified problems so that a canonical method becomes wrong while a physically correct method remains available.Trap records store predicted shortcut solvers and require a large shortcut–ground-truth separation; 62 traps were proposed by GPT-5.5.

Declarations

The authors report funding from two Chinese public-sector science programs and declare no competing interests; the benchmark is publicly available.

Declarations: The work was funded by the Natural Science Foundation of Guangdong Province and the Hebei Major Science and Technology Support Plan.The listed grants are 2025A1515012950 and 252S3701D.

Supplementary Tables

The supplementary analyses detail model coverage, robustness checks, execution-versus-prose failures, and tests of memorisation and paraphrase sensitivity.

Evaluated models: Thirteen models were evaluated as subjects, with exact API identifiers, access routes, and reasoning settings recorded for reproducibility.Nine core models yielded sixteen leaderboard configurations; domain-adapted models were treated as out-of-distribution references.
Failure mechanisms: −1.5 × 10−4 was returned by the executed protocol versus +1.5 × 10−4 in prose when a model dropped Fick’s-law sign information.The prose model wrote the signed formula but inserted a magnitude-taking step; the code implementation preserved the sign in 3/3 runs.
Failure mechanisms: Four case studies show prose failures from replacing iteration, truncating a long relation, dropping a required sign, or substituting a memorised one-liner.Execution and prose diverged mainly when models already possessed the correct method; code did not fix errors for every model.
Robustness and memorisation: No configuration showed a significant core-versus-numeric-variant accuracy gap after Holm correction, with the smallest corrected p-value equal to 0.34.Paraphrase effects likewise did not survive correction; the largest reported shift was DeepSeek-V4-pro at −3.7 points.

Supplementary Figures

The supplementary figures define the benchmark’s evaluation and construction protocols, characterize its 436-problem composition, and show that the construction method transfers across environmental domains.

Evaluation protocols: The evaluation protocols expose calculation by requiring models to return executable solve() functions under a fixed contract and sandboxed timeout.The model sees only the problem statement, while outputs must provide one value–unit entry per requested quantity without hard-coded answers.
Variant construction: The variant pipeline generates certified numeric and paraphrase variants from 436 core problems by rewriting controlled problem parameters and validating the resulting statements.Numeric perturbations freeze physical constants and hidden values while programmatically drawing new text-stated values.
Core-set composition: The core set contains 436 problems across ten categories, with atmospheric dynamics and thermodynamics together accounting for nearly half the corpus.Atmospheric dynamics contributes 116 problems and thermodynamics 89, while four categories contain 25 or fewer problems.
Cross-domain transfer: Cross-domain transfer preserves configuration ordering despite easier transferred problems, with all five configurations retaining their core-set rank.Transferred accuracy is 0.9 to 9.1 points above atmospheric-core accuracy, with Spearman ρ = 1.00.

Supplementary Notes

The supplementary notes illustrate how multi-stage environmental calculations propagate local mistakes into downstream quantities, making unit handling, intermediate transformations, and physical balances jointly important.

Example 1: cloud physics: The cloud-physics example converts a Gaussian droplet spectrum into concentration before balancing adiabatic production against condensational depletion.The required third raw moment is ⟨r^3⟩ = r̄^3 + 3r̄σ^2, rather than simply r̄^3.
Example 1: cloud physics: The calculation spans incompatible-looking units and ends with conversion from dimensionless saturation excess to percentage.It combines radii in µm, liquid-water content in g m−3, pressure in kPa, and concentration quantities in m−3 and kg−1.
Example 1: cloud physics: Only 4 of 16 configurations solve the cloud problem under the majority-of-three rule.An incorrect spectrum moment, missed unit conversion, or misassembled resistance changes every downstream quantity, with no intermediate target to localize the first error.

Example 2: a local derivation that breaks control-volume bookkeeping (boundary layer)

The boundary-layer example shows that even a short continuity derivation can fail when models do not track control-volume flux surfaces consistently.

Example 2: boundary layer: The correct derivation balances radial inflow through a cylindrical side area against compensating updraft through the top area.The setup uses an axisymmetric cylindrical control volume with radius R and boundary-layer depth Δz.
Example 2: boundary layer: The factor of two comes from the ratio of lateral cylindrical area to top area after common factors cancel.It is required by the geometry, not an optional coefficient.
Example 2: boundary layer: 7 of 16 configurations solve the problem, while 11 use w = u_rΔz/R and return 0.006 m s−1 instead of the 0.012 m s−1 reference value.These failures invoke continuity and use the correct inputs but omit the factor of two from the cylindrical side area.

Example 3: reconstructing a literature-specific empirical source function (atmospheric aerosols)

The aerosol examples test whether models can retrieve and execute a named empirical source function and preserve the distinction between mass- and number-weighted particle distributions.

Example 3: empirical source function: The sea-spray task requires integrating the Monahan et al. (1986) differential source function over a radius bin.Correct execution requires the empirical coefficients and exponents, radius convention, independent variable, and m−2-to-cm−2 conversion.
Example 3: empirical source function: 5 of 16 configurations solve the sea-spray problem, while 11 miss it, demonstrating a bottleneck in retrieving and correctly using a literature-specific parameterization.The problem-level matrix does not establish that all misses used the same alternative expression.
Example 4: size distributions: The particle-size problem requires separate mass- and number-weighted distributions despite using the same particles and diameter threshold.Particle mass scales as D^3, shifting the log-normal mean in ln D by 3σ^2 when converting representations.
Example 4: size distributions: The resulting fractions differ sharply: 2.3% by weight versus 65.0% by number below 1 µm.Small particles contribute little mass but can dominate particle number.

Supplementary Note S2: Trap failures

Trap tests alter one condition so a previously valid method no longer applies. Models often solve the parent problem but continue using its invalidated shortcut.

The trap requires replacing the parent shortcut after changing a condition that makes its method invalid.
Only 5 of 27 model-runs solve the trap, while 18 of 27 return the pre-registered small-deflection shortcut.These failures span 7 of 9 configurations, including frontier systems.
Failure transcripts assume midpoint arrival and constant-acceleration transverse displacement, despite the altered regime.

Example 2: Cartesian kinematics substituted for spherical geometry (atmospheric dynamics)

This trap tests whether models retain spherical-geometry terms when Cartesian kinematics is not valid. Most failing runs return the exact tangent-plane shortcut rather than the spherical result.

Only 6 of 27 model-runs solve the trap, while 12 runs from 6 configurations return the complete shortcut vector (0, 10, 10).The Cartesian shortcut drops every term proportional to tan ϕ/R.
The shortcut vector is exact for a different tangent-plane problem, not a rough numerical approximation to the spherical-geometry answer.

Example 3: a parent sign convention retained after the direction changes (thermodynamics)

The trap changes “east wind” to “eastward wind,” reversing the sign convention for the velocity. Most models solve it, but remaining failures preserve the parent convention and return the wrong sign.

The correct answer is −1.0 × 10^-3 °C s^-1, whereas retaining the parent convention gives +1.0 × 10^-3 °C s^-1.
Nineteen of 27 model-runs solve the trap.
Eight of 27 runs return the exact positive shortcut, preserving the parent sign convention after the wording changes.The arithmetic and advection equation remain intact, so the error is directional rather than computational.

Example 4: Earth-relative velocity returned for an inertial-frame target (atmospheric dynamics)

The trap asks for inertial-frame tangential velocity, but the parent formula returns Earth-relative wind. Models frequently preserve the parent target even while applying the correct angular-momentum principle.

Twelve of 27 model-runs solve the trap, while 4 runs across 3 configurations return the Earth-relative shortcut −5.469 m s^-1.
Absolute and Earth-relative velocities follow from the same conservation law but differ in reference frame and have opposite signs here.
The remaining failure is a definition and target-preservation error, not a failure to recall angular-momentum conservation.

Example 5: a layer-mean flux substituted for the required surface flux (boundary layer)

This example tests whether models distinguish a layer-averaged heat flux from the surface flux required by the Deardorff convective velocity scale. Most runs solve the trap, but some apply the correct formula to the wrong vertical measure.

Example 5: The correct Deardorff convective velocity scale is 3.532 m s−1.
Example 5: A linearly decreasing heat-flux profile makes the surface flux 1.34 K m s−1, twice the supplied layer average.The profile decreases from its surface value to zero at the top of the boundary layer.
How the models fall: Eighteen of 27 model-runs solve the trap.
How the models fall: Six runs from 3 configurations return exactly 2.803 m s−1, including a frontier configuration.These runs substitute the layer mean directly as the surface value.
How the models fall: These failures use the correct Deardorff formula and arithmetic but evaluate it with a quantity averaged over the wrong vertical measure.