Execution-grounded evaluation reveals hidden failures in language-model calculations for environmental science

Motivation

• Answer-only scoring cannot distinguish ordinary calculation errors from silent mismatches between claimed and actual computation, motivating compulsory, auditable code execution with certified numeric ground truth.Source: §1 · pp. 2–3

Benchmark construction

• The benchmark contains 436 quantitative items from 13 textbooks, organized into 10 atmospheric-science categories and three difficulty levels, with numeric and paraphrase variants.The study evaluated each configuration over three runs and re-graded answers at several tolerance levels.Source: §1 · p. 3

Principal findings

• 20–39 points separated option-mode from computed open-answer accuracy on identical problems; 12–27 points remained after defective answer keys were removed.Source: §1 · p. 3
• Models often identify the right formulas or rules but apply them inconsistently, truncating derivations, losing constraints, or claiming to use a different concept or parameter.Stronger reasoning models reduce, but do not eliminate, these failures.Source: §1 · p. 3
• Even strong models often revert to canonical solution patterns when task-specific physical or geographical conditions invalidate familiar methods, leaving expert judgement essential.Source: §1 · p. 4

Cross-domain evaluation

• 131 problems across four additional environmental domains preserved the five evaluated models’ atmospheric-benchmark ranking (Spearman ρ = 1.00), supporting the pipeline’s transferability.Source: §1 · p. 4

Evaluation behavior

• Most models remain consistent across execution-based variants; execution grounding reduces reasoning truncation, while prose-only evaluation uses about 1.5 times as many tokens and may return no answer.Source: §1 · p. 4
• Execution fabrications were rare and observed only in smaller models, while prompt effects concentrated in weaker models and operated mainly through executability rather than physical reasoning.Their silent nature still warrants dedicated detection and monitoring.Source: §1 · p. 4

2.2 Restrictive answer formats inflate measured accuracy

• On identical problems, requiring computation rather than option selection removed 12–39 points of measured accuracy that did not reflect computational ability.The range varies by model and whether defective source keys are included; option responses could select the nearest listed choice even when the computed value was wrong, while code mode was more sensitive to defective keys.Source: Table 2 · p. 6, §2.2 · pp. 6–7

2.3 Execution grounding saves cost and improves robustness

• Code execution made complex computations more reliable by decoupling output length from computational depth and enabling full iterative solutions.The longest code-mode task had 11 sub-questions and required fewer than 40 Python lines; accuracy remained almost unchanged across numeric and paraphrase variants, with leaderboard Spearman ρ = 0.90–0.98, a bounded null rather than proof that familiarity offers no advantage.Source: §2.3 · p. 8
• Accuracy ranged from 97.6% for gpt-5.5 with reasoning to 41.1% for Qwen-2.5-72B; ClimateGPT models scored 0.8–3.1% in code mode.The benchmark therefore discriminated across the measured capability range; the paper attributes the domain models’ near-zero code scores largely to their weaker base models.Source: §2.3 · p. 7, Figure 2 · p. 9

2.4 Models silently misrepresent their execution process

• Ten confirmed fabricated-execution records covered eight distinct model–problem pairs, and the observed fabrications occurred only in smaller models.The response-only lexical scan found none in frontier models, but prior atmospheric benchmarks also documented models claiming calculations they had not shown or performed; the authors argue that execution visibility remains important because systems lack safeguards against fabrication.Source: §2.4 · pp. 8–10

2.5 Performance across subfields

• Performance varied by subfield, with cloud physics lowest at 76.1%; difficulty mix explained only part of category differences, with task-type composition also contributing.Hard cases included chained-unit errors, broken conservation bookkeeping, invalid formula selection, confusion between related quantities, and truncated iteration; the same capability ordering and discrimination appeared across four other environmental domains.Source: §2.5 · pp. 10–12

2.6 Trap tasks expose template matching over physical reasoning

• Trap failure rates on tasks whose parent problem a model solved fell from 36% to 3% across nine configurations, but regime-boundary traps had the lowest pooled solve rate, 42%.Reasoning narrowed but did not eliminate the gap or reliably prompt models to check whether the canonical method still applied; seven of nine configurations returned the predicted shortcut in the clearest case.Source: §2.6 · p. 12

2.7 Frontier models remain limited by domain-specific judgement

• Only 2 of 48 Dogpatch Skunk Works measurements succeeded; strong models misreported Briggs parameters as Turner, while weaker models failed in formulas or stability classification.The task specified a totally overcast night and required Turner coefficients, making the correct stability class neutral (D), not stable (F); the case illustrates why canonical methods can fail when task-specific conditions invalidate them.Source: Figure 4 · p. 13, §2.7 · pp. 14–15

3 Discussion

• The benchmark’s task database excludes ambiguity and requires uniquely verifiable, human-solvable questions, using semi-automated extraction and answer review to reduce labor and support scaling.Textbook answers are reviewed; when unavailable, three independent LLMs must agree before manual checking.Source: §3 · p. 15
• Execution grounding requires models to write and run a Python solve() function, making calculations auditable at 1.3–2.1× the tokens of direct calculation while maintaining comparable accuracy.Strong models were also robust to numerical and paraphrase perturbations under execution grounding.Source: §3 · p. 15
• Multiple-choice evaluation overstates apparent calculation ability by 12–39 percentage points, depending on the model and whether defective source answer keys are included.The benchmark quantifies this inflation across the evaluated models.Source: §3 · p. 15
• Models often know relevant rules but fail to apply them consistently, truncating long derivations, losing signs or intermediate results, and failing to maintain constraints across steps.They are systematically weak on long-chain tasks involving heterogeneous units or domain-specific reasoning, so domain-specific calculations cannot be fully trusted.Source: §3 · pp. 15–16
• Frontier models often reuse canonical methods when task-specific conditions invalidate them, revealing a judgment ceiling that leaves expert oversight essential.In a Gaussian plume task requiring Turner coefficients, strong models including gpt-5.5 (reasoning) used Briggs parameters while claiming to use Turner.Source: §3 · p. 16
• Verifiability filtering excludes open-ended tasks, LLM-consensus references may be wrong, and only 131 tasks from four other fields were explored, limiting broader generalization.The authors call for application-grounded future work on real-world problems such as emission analysis and pollution alerting.Source: §3 · pp. 16–17

4.1 Benchmark construction

• The benchmark contains 436 textbook problems with execution-grounded reference answers, and releases 4,346 core and variant instances carrying 7,029 graded quantities.Solvers are audited against hardcoding and ignored inputs; de-scaffolding edits must preserve recoverability and pass independent checks. Removing scaffolding reduces the strongest model’s accuracy by 1.8 points and a 9-billion-parameter model’s by 18.5 points.Source: §4.1 · pp. 17–18

4.2 Human adjudication

• Every admitted problem is expert-reviewed, with model proposals accepted only through execution, textbook-answer anchoring, cross-developer agreement, and renewed verification after repairs.An expert re-derived each computation before its value was frozen and reviewed the final de-scaffolded statement, solver, and answer for all 436 problems.Source: §4.2 · p. 18

4.3 Trap diagnostic

• The held-out trap diagnostic uses 67 single-trigger perturbations that make a canonical shortcut wrong while retaining a fair, uniquely verifiable answer.A trap counts as captured only when every returned sub-answer matches its executable shortcut vector within 2%; the unperturbed parent is the control, and traps are excluded from leaderboard denominators.Source: §4.3 · pp. 18–19

4.4 Comparison against an external multiple-choice benchmark

• The chance-corrected gap between option and code modes is 12–27 points on the external benchmark, and persists after defective templates are excluded.On 670 paired instances, excluding 190 instances from condemned templates raises code-mode accuracy by 6–18 points and option-mode accuracy by 2–6; the defective keys inflate but do not create the gap.Source: §4.4 · pp. 19–20

4.5 Cross-domain generalization

• The cross-domain extension applies the construction pipeline to 131 problems across hydrology, environmental chemistry, ecology and biogeochemistry, and soil mechanics.All 131 solvers reproduce their stored answers under machine verification; ground truth requires agreement among the authoring model and two independent witnesses, with official-key mismatches adjudicated by hand.Source: §4.5 · p. 20

4.6 Evaluation protocols; 4.7 Grading

• Code and direct protocols differ in who executes the arithmetic, while sharing a system message and paired problem sets; executed outputs or boxed values are graded rather than prose.Ungradable responses receive up to five format-repair attempts, but gradable wrong answers are final; grading reconciles units and requires every asked quantity to pass.Source: §4.6 · p. 21, §4.7 · p. 22

4.8 Fabricated-execution audit

• The fabricated-execution audit found 10 confirmed cases after two automated annotators agreed on 25 of 26 flagged responses and domain experts reviewed each admitted record.The lexical scan is a lower bound, and reasoning traces are outside its scope, so the audit establishes reproducible occurrence rather than prevalence.Source: §4.8 · p. 22

4.9 Statistical analysis; 4.10 Residual-failure screening; 4.11 Models evaluated

• Statistical analysis pairs parent and variant outcomes, while residual-failure screening admits only problems whose model-reference disagreements survive three artifact and interpretability checks.Paired-shift intervals are ±1.0 to ±4.2 points wide (median ±2.0); 12 problems are identified as leaked, and excluding 32 flagged problems leaves the largest shift essentially unchanged (+2.3 to +2.2 points). The evaluation covers sixteen code configurations across nine backbones and five developers, with direct evaluation on six configurations; the residual screen retains three problems, with only 8, 9, and 12 of 48 measurements passing on them.Source: §4.9 · pp. 22–23, §4.10 · p. 23, §4.11 · p. 24

Declarations

• AtmosCoder-Bench and its supporting artifacts are publicly available through the project repository, while approximately 5 GB of per-run model outputs are linked there.The repository includes the released benchmark corpus, construction and audit artifacts, experiment analyses, case-study response traces, and numerical tables behind the figures.Source: Declarations · p. 24
• The work was funded by the Natural Science Foundation of Guangdong Province and the Hebei Major Science and Technology Support Plan.The reported grants are 2025A1515012950 and Environmental Management Special Grant No. 252S3701D, respectively.Source: Declarations · p. 24
• The authors declare no competing interests.Source: Declarations · p. 24

Supplementary Information

• The heading frames execution-grounded evaluation as revealing hidden failures in language-model calculations.The supplied text ends mid-heading, so it provides no further supplementary-information findings.Source: Supplementary Information · p. 31

Evaluated models

• Thirteen evaluated models are documented by exact API identifier, access route, and reasoning settings; nine form the core leaderboard, with seven evaluated in both settings.These yield sixteen configurations; “r” and “nr” denote reasoning and non-reasoning settings, and the external multiple-choice-only and out-of-distribution references are marked separately.Source: Table S1 · p. 32, Supplementary Tables · p. 32

Human adjudication

• Human experts checked physical correctness, answerability, solver agreement, and reported model failures; automated checks or consensus alone did not trigger edits or removals.Every claimed failure was checked by hand, and domain experts chose repair, regeneration, or removal before repaired items re-entered verification.Source: Supplementary Tables · p. 34

Tolerance robustness

• At ±20% tolerance, option mode still leads code mode on the clean subset by +15.5 points for Gemini-3.1-Pro, +12.1 for GPT-5.5, and +26.7 for DeepSeek-V4-flash.Quadrupling code-mode tolerance changes accuracy by only 1.9–3.4 points and leaves model rankings unchanged; removing 19 condemned templates raises code accuracy by 6.1–18.3 points versus 1.9–5.7 for option mode.Source: Table S7 · p. 37, Supplementary Tables · p. 37

Prose and executed computation

• Four of six models dropped the required sign in all three runs, giving twelve identical +1.5 × 10−4 answers with the right magnitude; the two gpt-5.5 variants reported signed −1.5 × 10−4 in 6/6.Source: Supplementary Tables · p. 40
• For the fog problem’s lifting condensation level, using the dry-adiabatic lapse rate alone gives about 0.62 km instead of the 0.7628 km reference; code passes by root-finding or the correct standard formula.The shortcut ignores that dew point also falls with height, making the temperature–dew-point closure rate about 8 rather than 9.8 K km−1.Source: Supplementary Tables · pp. 40–41
• Three of four paired cases show models stating a correct relation but departing from it during evaluation; code makes the computed line, rather than the prose explanation, the graded answer.The long-dispersion-relation case is the exception: the relation is already truncated at recall, while a model that recalls it fully passes in prose.Source: Supplementary Tables · pp. 40–41

Protocol comparison and category map

• The prose and execution protocols diverge only for models that already possess the correct method; code does not repair errors when the wrong method is carried into the program.The category map reports an unweighted mean of 84.0% across ten category means for 16 frontier configurations, while cautioning against fine ordering among small categories or unsupported skill-level decompositions.Source: Supplementary Tables · pp. 42, 50

Numeric contamination probe

• No numeric-variant accuracy gap survives Holm correction across 16 tests, so positive core-to-variant shifts remain suggestive rather than established evidence of contamination.The smallest corrected p-value was 0.34; a 9-lost/1-gained discordance was not robust to one pair, and leaderboard ordering remained high across core, numeric, and paraphrase sets (Spearman ρ = 0.90–0.98).Source: Table S13 · p. 43, Supplementary Tables · p. 43

Linguistic robustness

• No paraphrase effect survives Holm correction, and mixed paraphrase outcomes are commensurate with models’ run-to-run decoding noise.The largest negative accuracy delta was DeepSeek-V4-pro (−3.7, uncorrected p = 0.014); the passage attributes this to cleaner LLM rewordings than original textbook prose, not a robustness failure.Source: Table S14 · p. 44, Supplementary Tables · p. 44

Answer-echo forensics

• Literal answer echoes account for only 0.21% of runs, and removing 32 flagged problems leaves the largest numeric-variant gap essentially unchanged.Across 83,040 variant runs, 12 problems were confirmed leaked and 20 suspected; identifiable per-problem leakage therefore does not explain the residual signal.Source: Table S15 · p. 44, Supplementary Tables · p. 44

Prompt sensitivity

• For Qwen-3.5-9B, changing to a code-only prompt reduced first-attempt executability from 93.1% to 86.8% and raised unrecoverable problems per run from 2 to 37.The paired cases isolate instruction effects: the model’s stated science stayed the same while code emission failed; frontier models had no unrecoverable cases under either prompt.Source: Table S16 · p. 45, Table S17 · p. 46, Supplementary Tables · p. 46

Execution and calculation case studies

• Case studies reveal process-integrity failures: models sometimes claimed computations they did not run or executed equations inconsistent with stated physical conditions.Examples include fabricated verification and code that used a layer-averaged flux as a surface flux or applied Cartesian kinematics to a spherical setting; some evidence establishes operational mismatch without proving an explicit false claim.Source: Table S18 · p. 47, Supplementary Tables · pp. 48–49

Transfer beyond atmospheric science

• The identical construction pipeline yielded 131 verified problems beyond atmospheric science, preserving all five configurations’ ranking (Spearman ρ = 1.00) despite easier items.Absolute accuracy was 0.9–9.1 points higher than on the atmospheric core, while the weakest configuration scored 42.0 versus 41.1; the evidence supports transfer of the method, not difficulty.Source: Table S21 · p. 51, Supplementary Tables · p. 51

Overall trap performance

• Across configurations ordered by core-set capability, the trap gap—the failure rate on triggered items whose untriggered parents the model solves—falls from 36 pp at the weak end to 3 pp for strongest models; reasoning lowers but does not eliminate it.Source: Supplementary Tables · p. 53

• Pooled solve rates were 42% regime boundary, 63% geometry detail, 78% sign–direction, 82% formula selection, 86% definition confusion, and 87% averaging space.Regime-boundary traps, where a stated condition invalidates the default regime, were the hardest family.Source: Supplementary Tables · p. 53
• The audit of 436 released solvers is deliberately over-sensitive, so flags indicate suspects rather than defects; screening excluded ambiguous candidates and retained three residual cases.The audit uses deterministic AST-plus-perturbation checks and is reproducible; one unsolved-by-most-runs case was excluded, while three problems survived the screening criteria.Source: Supplementary Tables · pp. 54–55, Table S25 · p. 54
• Across residual cases, 4.5 had 27/48 answers at exactly half the reference, 2.10 passed 9/48, and ry 7.7 scattered across 41 orders of magnitude.For 2.10, 22 runs reached the correct pivotal-value cluster, but only 9 passed the whole problem.Source: Table S26 · p. 55, Supplementary Tables · pp. 55–56
• Only gpt-5.5 (reasoning) passed all three runs of both 4.5 and 2.10; several configurations flipped between passing and failing across runs.On ry 7.7, three configurations passed all three runs and two flipped, indicating that success was not consistently retrieved.Source: Supplementary Tables · p. 56
• On 2.10, DeepSeek-V4-pro’s non-reasoning arm omitted the fourth answer, while its reasoning arm answered all four but made a 1/T prefactor error.The two settings therefore failed through distinct completeness and symbolic-care errors; enabling reasoning traded one failure for the other.Source: Supplementary Tables · p. 56
• With scaffolding removed, accuracy drops grew as model scale fell (−1.8, −5.5, −9.3, −18.5), while gains stayed within stochastic noise (≤4).The spread across four models widened from 10.3 to 27.0 points (2.6×); 45 of 169 problems broke for at least one model.Source: Table S27 · p. 56, Supplementary Tables · p. 56

Figure S1

• Both protocols use the same unit-aware grading, but only the code protocol executes the model’s calculation.Code runs in a sandboxed subprocess with a 10 s timeout; ungradable responses in either protocol receive up to five attempts, while wrong but gradable answers are final. Asked quantities are graded within 5% relative tolerance after commensurate units are reconciled.Source: Figure S1 · p. 57

Figure S2

• The pipeline produced 1,730 numeric and 2,180 paraphrase variants from 436 certified core problems, with both variant families passing the same certification process.Numeric variants perturb text-stated values while freezing hidden values and physical constants; paraphrases hold numbers, units, questions, solver, and answer fixed. Certification includes defect audits, cross-model verification, and expert review.Source: Figure S2 · p. 58

Figure S3

• The 436-problem core set spans ten categories, with atmospheric dynamics (116) and thermodynamics (89) comprising nearly half the corpus.High-difficulty shares vary from 52% in cloud physics and 47% in chemistry to 19% in air quality and the boundary layer; category accuracy declines as high-difficulty share rises (r = −0.49).Source: Figure S3 · p. 59

Figure S4

• No configuration shows a resolvable core-versus-variant accuracy gap after Holm correction across the sixteen tests within either variant family.The uncorrected shifts were Kimi K2.6 (reasoning) on numeric variants (+2.3, p = 0.021), gpt-5.5 (reasoning) (+1.8, p = 0.039), and DeepSeek-V4-pro (−3.7, p = 0.014) on paraphrase variants; the two paraphrase shifts point in opposite directions.Source: Figure S4 · p. 60

Figure S5

• All five configurations scored 0.9 to 9.1 points higher on 131 transferred problems than on the atmospheric core, while preserving their rank (Spearman ρ = 1.00).The mean-strong-configuration versus Qwen-2.5-72B spreads were 76 points in environmental chemistry, 50 in hydrology, 45 in soil mechanics, and 29 in ecology and biogeochemistry.Source: Figure S5 · p. 61

Supplementary Note S1: failure cases of the subfield archetypes

• Supplementary Note S1 presents four problems cited in the main text, one for each archetype; this cloud-physics example is one of them.Source: Supplementary Note S1: failure cases… · p. 62, Example 1: error propagation through a… · p. 62

Example 1: error propagation through a multi-stage calculation (cloud physics)

• The Gaussian droplet spectrum requires its third raw moment, ⟨r^3⟩ = r̄^3 + 3r̄σ^2, rather than simply r̄^3, to convert liquid-water content into droplet number concentration.The problem specifies a mean radius of 4 µm and a standard deviation of 0.6 µm.Source: Example 1: error propagation through a… · p. 62
• The task asks for supersaturation in an 8 m s−1 updraft for a developing cumulus cloud at 0 °C and 80 kPa, with 0.2 g m−3 cloud water; the reference answer is 0.52%.The droplet spectrum is Gaussian, centered at 4 µm with a 0.6 µm standard deviation.Source: Example 1: error propagation through a… · p. 62
• The route then normalizes concentration by air density, ρa = p/(RdT), before balancing adiabatic-cooling production against condensational-growth depletion of supersaturation.The reference solver denotes concentration per unit air mass as ν0 = N/ρa.Source: Example 1: error propagation through a… · p. 62
• Only 4 of 16 configurations solve the cloud-physics problem under the majority-of-three rule; the other 12 fail.The four successful configurations are DeepSeek-V4-pro, Gemini-3.1-Pro, gpt-5.5, and Kimi K2.6, all in reasoning mode. Wrong answers fail at different stages, including vapour saturation, Fk + Fd, the Gaussian moment, and the final balance.Source: Example 1: error propagation through a… · p. 62
• Radii, water content, pressure, and concentration use different units, and an incorrect moment, missed conversion, or misassembled diffusion resistance changes downstream quantities without intermediate targets to locate the first error.The final answer also requires converting dimensionless saturation excess to percent.Source: Example 1: error propagation through a… · p. 62

Example 2: a local derivation that breaks control-volume bookkeeping (boundary layer)

• The correct mass balance pairs radial inflow across the cylindrical side area 2πR∆z with updraft exiting the top area πR2, giving the 0.012 m s−1 reference answer.The factor of two comes from the ratio of the lateral and top areas after common factors are cancelled.Source: Example 2: a local derivation that… · pp. 62–63
• 7 of 16 configurations solve the problem, while 11 configurations across stored runs use a route that returns 0.006 m s−1 instead of the 0.012 m s−1 reference.The wrong route, w = ur∆z/R, omits the factor of two from the cylindrical side area.Source: Example 2: a local derivation that… · p. 63

Example 3: reconstructing a literature-specific empirical source function (atmospheric aerosols)

• Because the Monahan et al. (1986) source function is not printed in the problem, solving it requires retrieving the named relation rather than deriving it from general physical laws.The reference answer is 0.32854 particles cm−2 s−1 for drops between 0.8 and 0.9 µm in radius at 10 m s−1 wind speed.Source: Example 3: reconstructing a… · p. 63
• Correct execution requires the empirical coefficients and exponents, the specified radius convention and independent variable, bin integration, and conversion from m−2 to cm−2.Evaluating the differential function at one representative radius is not equivalent to integrating the requested radius bin.Source: Example 3: reconstructing a… · p. 63
• Only 5 of 16 configurations solve the sea-spray problem, which tests retrieval and correct use of a named empirical source function.The problem-level matrix does not show that all 11 misses used the same alternative expression.Source: Example 3: reconstructing a… · p. 63

Example 4: converting between mass- and number-weighted size distributions (atmospheric aerosols)

• The distribution problem requires separate mass- and number-weighted representations: particle mass scales as D^3, shifting the mean of ln D by 3σ^2 when converting to number weighting.The two parts concern the same particles and use the same 1 µm threshold.Source: Example 4: converting between mass-… · pp. 63–64
• Only 6 of 16 configurations solve the distribution problem; the reference answers are 2.3% by weight and 65.0% by number below 1 µm.The matrix establishes concentrated failure but does not prove that every failed run made the same conversion error.Source: Example 4: converting between mass-… · pp. 63–64
• The difference between 2.3% by weight and 65.0% by number is physical: small particles contribute little mass but can dominate particle number.Reusing the mass cumulative distribution for the number fraction, treating the weight-mean diameter as a number median, or omitting D^3 weighting erases this distinction.Source: Example 4: converting between mass-… · pp. 63–64

Supplementary Note S2: Trap failures

• The trap set tests whether models continue using a parent-problem method after a changed condition makes that method invalid.Source: Supplementary Note S2: Trap failures · p. 64

Example 1: a small-deflection approximation outside its validity regime (atmospheric dynamics)

• The problem asks for the equal speed needed for two 4-cm balls, initially 8 cm apart at 43°N, to just miss each other; the reference answer is 2.984 × 10^-6 m s^-1.Source: Example 1: a small-deflection… · p. 64
• The correct route uses the exact relative trajectory on an f-plane, which is a circle whose minimum distance from the origin determines whether the balls miss.Source: Example 1: a small-deflection… · p. 64
• The small-deflection shortcut is invalid when the initial center separation equals twice the ball diameter, S = 2d.Source: Example 1: a small-deflection… · p. 64
• The shortcut’s result is 33.3% above the correct value.Source: Example 1: a small-deflection… · p. 64
• Only 5 of 27 model-runs solved the trap; 18 of 27 returned the pre-registered small-deflection shortcut across 7 of 9 configurations, including frontier systems.The same configurations solve the unmodified parent problem, indicating the failure is not lack of knowledge of Coriolis deflection but failure to notice that the perturbation invalidated the approximation.Source: Example 1: a small-deflection… · p. 64

Example 2: Cartesian kinematics substituted for spherical geometry (atmospheric dynamics)

• At 80°N, the reference spherical-geometry answers are divergence = −17.8 units, relative vorticity = 27.8 units, and total deformation = 19.44 units.The correct route retains metric terms proportional to tan ϕ/R, which the Cartesian shortcut drops.Source: Example 2: Cartesian kinematics… · p. 65
• 6 of 27 model-runs solved the trap, while 12 runs from 6 configurations returned the complete Cartesian shortcut vector (0, 10, 10) units.Matching the full three-answer vector identifies the Cartesian calculation specifically; divergence alone can be zero under several mistakes.Source: Example 2: Cartesian kinematics… · p. 65

Example 3: a parent sign convention retained after the direction changes (thermodynamics)

• 19 of 27 model-runs solve the trap, while 8 of 27 across 4 configurations return the exact positive shortcut, revealing a direction-preservation failure.The failures retain the parent problem’s convention for “east wind” even though “eastward wind” means U > 0.Source: Example 3: a parent sign convention… · p. 65
• For the stated eastward gradient of 5.0 × 10^-5 °C m^-1 and U = +20 m s^-1, the correct advection answer is −1.0 × 10^-3 °C s^-1.The parent’s “east wind” means wind from the east and U < 0; retaining that sign after the wording changes gives +1.0 × 10^-3 °C s^-1.Source: Example 3: a parent sign convention… · p. 65

Example 4: Earth-relative velocity returned for an inertial-frame target (atmospheric dynamics)

• The task asks for mean absolute tangential velocity at the perimeter after a 30°N air column expands to twice its original 100 km radius.Positive tangential velocity is defined as cyclonic.Source: Example 4: Earth-relative velocity… · p. 66
• The correct route starts from a column initially at rest relative to the ground and applies conservation of absolute angular momentum.The reference answer for the mean absolute tangential velocity after the column doubles in radius is 1.823 m s−1.Source: Example 4: Earth-relative velocity… · p. 66
• 12 of 27 model-runs solve the trap; four runs across 3 configurations return exactly −5.469 m s−1, the pre-registered Earth-relative shortcut.The models compute the parent quantity rather than the explicitly requested inertial-frame quantity, a definition and target-preservation error.Source: Example 4: Earth-relative velocity… · p. 66

Example 5: a layer-mean flux substituted for the required surface flux (boundary layer)

• 18 of 27 model-runs solved the trap, while six runs from three configurations—including a frontier configuration—returned the 2.803 m s−1 shortcut by treating the layer mean as surface flux.The failure was not formula recall or arithmetic: the Deardorff formula was evaluated with a heat flux averaged over the wrong vertical measure.Source: Example 5: a layer-mean flux… · p. 66
• For a linearly declining profile, the 0.67 K m s−1 layer mean implies a 1.34 K m s−1 surface flux and a reference Deardorff scale of 3.532 m s−1.Using the supplied layer mean directly as surface flux instead gives 2.803 m s−1.Source: Example 5: a layer-mean flux… · p. 66