PaperlayerView exact revision on arXiv

Source-linked AI summary

How Reliable Are Large Language Models for Soil Organic Carbon Assessment? An Australian Benchmark

Yingxuan Cheng, Tong Li, Yayong Li, Yuchuan Fan, You-Gan Wang, Jinran Wu, Scott Chapman, Geoffrey J. McLachlan

arXiv:2610.18201v1cs.PFcs.AIcs.CY

TL;DR

Field-referenced SOC estimates from LLMs require validation that existing general benchmarks cannot provide. This paper benchmarks seven configurations on concentration estimation, stock calculation, and paddock comparison, finding distinct reliability challenges across these tasks and limits on what repeated-call results establish.

  • Problem

    Existing LLM benchmarks do not establish whether outputs are correct for field-referenced SOC estimates, which require specified reference quantities and conditions.

    Source: §1 · p. 2

  • Method

    Seven LLM configurations were evaluated on concentration estimation, stock calculation, and pairwise paddock comparison against measured SOC records or deterministic calculations.

    Source: §1 · p. 2

  • Results

    Across the three tasks, added soil attributes reduced concentration error but high-SOC underprediction and local failures remained; stock calculations were vulnerable to equivalent unit changes, and similar-paddock comparisons were less accurate and sometimes order-sensitive.

    Source: §5 · p. 15

  • Takeaways & Limitations

    SOC assessment requires separate checks of agreement with measured concentrations, unit-consistent stock calculations, and discrimination between sites.

    Source: §5 · p. 15

    References covered sampled 0–10 cm areas rather than whole paddocks or deeper profiles, and repeated-call intervals do not estimate performance on new farms or scenarios.

    Source: §4.6 · p. 15

Abstract

from arXiv · show

Soil organic carbon (SOC) is an indicator of agricultural soil condition and carbon storage, but numerical outputs from large language models (LLMs) require validation before use in monitoring or management. We evaluated seven general-purpose LLM configurations on three direct-response tasks: estimating 0--10 cm SOC concentrations for 308 Australian paddocks under two information conditions, calculating SOC stock in 120 matched scenarios, and comparing 300 paddock pairs. References were measured SOC records or deterministic calculations; response validity, repeated-call dispersion and estimated API cost were assessed separately. Additional measured soil attributes reduced concentration mean absolute error by 13--62\% across configurations, although high-SOC underprediction and paddock-specific errors persisted. Six configurations were highly accurate in stock calculations using equation-native units without coarse fragments, but accuracy declined with physically equivalent alternative units. Pairwise accuracy was 90.5--94.6\% overall and 59.1--70.9\% for 52 closely separated pairs; some configurations were sensitive to presentation order. Nearly all outputs were parseable, without necessarily being correct. The benchmark identifies three distinct requirements for AI-mediated SOC assessment: agreement with measured concentrations, unit-consistent stock calculations and reliable discrimination between sites. These findings support task-specific validation and auditable calculations, rather than unverified use of LLM outputs as ecological evidence.

1 Introduction

Field-referenced SOC outputs from LLMs need SOC-specific validation because existing benchmarks cannot establish whether estimates remain correct under field conditions. This study tests concentration estimates, stock calculations, and paddock comparisons as distinct operations with different references and error consequences.

1 Introduction

  • SOC’s usefulness depends on soil context, sampling depth, intended use, and reference system; acceptable error and useful discrimination vary by application.SOC alone does not establish ecosystem services or management impacts.Source: §1 · p. 1
  • LLM benchmark findings cannot be transferred directly to field-referenced SOC estimates, which require a specified reference quantity and tests of output correctness.Results on regression and geographic tasks depend on examples, contextual inputs, and spatial information.Source: §1 · p. 2
  • Three monitoring risks are disagreement with measured concentration, inconsistent stock calculations under equivalent units, and unreliable ordering of paddocks with similar SOC.Because these operations have different references and error consequences, a single aggregate score could obscure their limitations.Source: §1 · p. 2
  • The study benchmarks seven LLM configurations across three direct-response tasks and uses measured SOC records or deterministic calculations as references.Calls exclude retrieval, external calculators, and conversation history; the study provides a baseline for future tool-assisted workflows, not a replacement for measurement or a composite model ranking.Source: §1 · p. 2

2 Methods

The empirical references came from 308 Australian agricultural sites and represented sampled 0–10 cm areas, not whole paddocks. Each paddock’s reference concentration was computed from its retained analytical records.

2 Methods

  • The dataset resampled 308 agricultural sites across six Australian states during 2022–2025, revisiting locations surveyed in 2009–2012.The study targeted ten cores per site, generally within a 25 × 25 m grid, with larger areas at some Queensland rangeland sites.Source: §2.1 · p. 2
  • Each paddock reference was the unweighted arithmetic mean of its retained 0–10 cm analytical records, expressed in g C kg-1 on an oven-dry soil-mass basis.The 1,213 retained records covered all 308 sites; reference concentrations ranged from 1.1 to 93.0 g C kg-1, with a median of 18.0 g C kg-1.Source: §2.1 · p. 2
  • Laboratory documentation describes high-temperature oxidative combustion, but the released table lacked a record-specific method field, so methods could not be independently verified for each record.No inter-method conversion was applied.Source: §2.1 · p. 2
  • Figure 1 locates the 308 paddock sampling points across six states; the points do not mark paddock boundaries or encode SOC concentration.The locations are dataset-reported sampling locations.Source: Figure 1 · p. 3
  • Table 1 summarizes paddock-level reference SOC concentrations and retained analytical records for the 0–10 cm layer by state.Overall SOC statistics were calculated directly across all 308 paddocks rather than by averaging state-level summaries.Source: Table 1 · p. 3, §2.1 · p. 3

2.2 Model configurations and evaluation protocol

Seven configurations from six providers were evaluated using separate API calls under recorded settings. The benchmark planned 145,040 responses, and provider-side model versions could not be independently verified.

2.2 Model configurations and evaluation protocol

  • The benchmark evaluated seven configurations from six providers, defining each configuration by the requested model ID and recorded settings.Provider-side versions could not be independently verified.Source: §2.2 · p. 4
  • There were 145,040 planned responses across 616 T1 items, 120 T2 items, and 300 T3 pairs.Each configuration had 20 planned responses per T1 or T2 item and per T3 pair, with T3 responses split equally between presentation orders.Source: §2.2 · p. 4
  • The table lists the evaluated model configurations, while token-limit definitions differed among provider interfaces.Thus, identical numerical token limits did not imply identical visible-output budgets.Source: Table 2 · p. 4, §2.2 · p. 4
  • Each response used a separate API call without conversation history, retrieval, or tool access.Temperature, top-p, and seed were unspecified; reasoning controls differed across configurations, and the settings do not establish equivalent internal computation.Source: §2.2 · p. 4

2.3 Quantitative SOC tasks

The methods test three complementary operations: concentration agreement with measurements, stock calculations under unit and fragment conditions, and pairwise paddock ordering. Their distinct references, metrics, and statistical contrasts were not combined into a single score.

2.3.1 T1: SOC concentration estimation

  • T1 estimated 0–10 cm SOC concentration under two nested information conditions, with the richer condition adding soil properties and historical land use.Additional fields were evaluated together rather than through individual-field ablations; four supervised baselines contextualized estimates using five-fold spatial cross-validation.Source: §2.3.1 · p. 4, Figure 2 · p. 5

2.3.2 T2: SOC stock calculation

  • T2 tested a supplied stock equation on 30 synthetic base scenarios, crossing equivalent unit representations with conditions with or without coarse fragments to yield 120 items.The coarse-fragment correction used a dry-mass fraction; unit and fragment contrasts tested different operations.Source: Figure 2 · p. 5, §2.3.2 · p. 5

2.3.3 T3: Pairwise SOC comparison

  • T3 assessed which of two paddocks had higher reference SOC, using 300 sampled pairs and balancing the higher-SOC paddock’s presentation as A or B.Accuracy was also analyzed by continuous SOC separation and correct-answer position, with a sensitivity analysis separating 248 pairs from 52 pairs at a nominal separation threshold.Source: Figure 2 · p. 5, §2.3.3 · p. 6

2.3.2 T2: SOC stock calculation

  • A T2 response counted as correct when its unrounded absolute relative error from the deterministic reference was no more than 2%.Accuracy was averaged across planned responses per item and then equally across items; repeated-call intervals were conditional on fixed benchmark items.Source: §2.3.2 · p. 6

2.4 Response validity, repeated-call variability and estimated cost

Response validity, repeated-call variability, and estimated API cost were assessed separately. Scoring rules differed by task, and the variability measures captured task-specific response spread rather than correctness.

2.4 Response validity, repeated-call variability and estimated cost

  • Only valid finite predictions contributed to T1 errors, while invalid and unavailable responses counted as incorrect for T2 and T3.Strict JSON parsing and prespecified fallback acceptance were recorded separately, without reference-assisted repair; answers were scored as finite numbers for T1 and T2 or A/B for T3.Source: §2.4 · pp. 6–7
  • T1 dispersion was the IQR of valid predictions per configuration–paddock–condition cell; T2 dispersion was the IQR of valid finite answers per configuration–item, scaled by absolute reference stock.Source: §2.4 · p. 7
  • T3 disagreement was calculated within each configuration–pair–presentation-order cell, with cell distributions equally weighted and cells with fewer than two valid responses unavailable.Zero disagreement did not establish correctness, and task-specific variability measures were not combined into a common stability score.Source: §2.4 · p. 7
  • API costs were rate-based estimates from interpretable provider-reported token usage across attempts, including retries, rather than reconstructed invoices.Unknown usage retained a separate cost reservation; fixed input and output rates were those recorded on 29 September 2026.Source: §2.4 · p. 7

2.5 Statistical analysis

Statistical uncertainty was framed around repeated responses on fixed benchmark items, with separate procedures for descriptive summaries, intervals, and configuration contrasts.

2.5 Statistical analysis

  • T2 and T3 intervals quantify repeated-call uncertainty conditional on benchmark items and references, not performance on new monitoring sites; T1 was descriptive without intervals or tests.Source: §2.5 · p. 7
  • Bootstrap intervals resampled 20 binary outcomes per T2 item and 10 per T3 pair–presentation-order cell across 10,000 replicates, with deviations rescaled by n/(n −1).Pointwise 95% intervals used the 2.5th and 97.5th percentiles of the rescaled distributions.Source: §2.5 · p. 7
  • The 21 prespecified T3 configuration-accuracy contrasts used normal-approximation tests with Holm-adjusted P values; adjustment did not alter pointwise intervals.A secondary model gave conditional GAP predictions and position contrasts with random effects set to zero; its 95% Wald intervals were conditional on estimated penalties, and no cross-task composite score was calculated.Source: §2.5 · p. 7

3 Results

Across three SOC tasks, added information reduced concentration error, stock-calculation accuracy depended on unit representation, and pairwise accuracy fell sharply for closely separated paddocks. High-SOC underprediction and repeated-call variability remained, despite generally high response validity.

3.1 T1: SOC concentration estimation

  • Additional information reduced T1 MAE by 1.51–6.20 g C kg-1 (13–62%) across configurations; T1b MAE was 8.78–11.51 versus 3.62–10.00 g C kg-1 under T1a.The reduction was heterogeneous: errors sometimes increased for individual paddocks, and high-SOC underprediction persisted.Source: §3.1 · pp. 7–8, Table 4 · p. 8, Figure 3 · p. 9

3.2 T2: SOC stock calculation

  • Under equation-native units without coarse fragments (U0F0), six configurations achieved 92.8–100.0% stock-calculation accuracy, while Mistral Medium 3.5 achieved 58.3%.Overall T2 accuracy ranged from 38.9% to 89.0%.Source: §3.2 · p. 8, Table 4 · p. 8
  • Unit re-expression reduced T2 accuracy by 21.6–45.2 percentage points for six configurations; coarse-fragment contrasts ranged from -5.3 to +1.3 points for five configurations.GPT-5.6 Terra’s coarse-fragment contrast was -10.3 points and Mistral Medium 3.5’s was -39.3; overall accuracies of 76.1–78.2% concealed distinct condition profiles.Source: §3.2 · p. 8

3.3 T3: Pairwise SOC comparison

  • Across 300 paddock pairs, T3 accuracy ranged from 90.5% to 94.6%; six configurations scored 93.5–94.6%, versus 90.5% for Mistral Medium 3.5.Eighteen of 21 between-configuration contrasts were significant after Holm correction, despite a 1.1-percentage-point range among the six higher-scoring configurations.Source: §3.3 · p. 8, Table 4 · p. 8
  • T3 accuracy was 97.0–99.6% for 248 pairs above the nominal separation threshold, but 59.1–70.9% for the remaining 52 closely separated pairs.Those 52 pairs were 17% of the sample and accounted for most errors; Mistral Medium 3.5 had an estimated presentation-order effect of +18.1 percentage points (95% interval 9.8–26.4).Source: §3.3 · p. 8

3.4 Response validity, repeated-call variability and estimated cost

  • Of 145,040 planned responses, 145,011 were valid (99.98%), with at least 99.7% validity in each configuration–task cell.Repeated-call dispersion varied by configuration and condition; T2 relative IQRs were often near zero but had long upper tails, especially for Claude Sonnet 5 and Mistral Medium 3.5.Source: §3.4 · pp. 9–10
  • Estimated API cost was US$174.64: US$86.30 for T1, US$29.85 for T2, and US$58.49 for T3.Configuration totals ranged from US$7.30 for DeepSeek V4 Pro to US$80.92 for Grok 4.6; these were standardized provider-rate estimates, not invoiced costs.Source: §3.4 · p. 10

4 Discussion

The discussion finds that SOC reliability varies by task and information representation: added soil measurements improve concentration estimates but do not remove bias, equivalent unit changes disrupt stock accuracy, and close paddock comparisons remain difficult. It recommends checks matched to the intended use while noting that management effects were not tested.

4.1 Measured inputs improve SOC indicator agreement but do not eliminate bias

  • Added soil attributes improved concentration estimates, but high-SOC underprediction and local errors remained, and the blockwise design could not isolate individual predictor effects.The improvement shows responsiveness to supplied observations, not independent measurement of SOC; the discussion cautions that local status-class changes and management outcomes were not assessed.Source: §4.1 · pp. 10–11

4.2 Equivalent units expose a failure of stock-indicator consistency

  • Six configurations lost approximately 22–45 percentage points of stock-calculation accuracy under physically equivalent unit re-expression, despite near-ceiling accuracy in native units.Unit changes should preserve the same carbon quantity; coarse fragments, by contrast, alter the reference stock and require correct dry-mass-fraction treatment.Source: §4.2 · pp. 12–13

4.3 SOC discrimination requires separation- and order-aware evaluation

  • High mean pairwise accuracy concealed weaker discrimination for closely matched paddocks, while reversing A/B order affected some responses, notably Mistral Medium 3.5.The nominal GAP split is specific to this sample, reference orderings may carry uncertainty, and T3 compared different paddocks rather than change over time.Source: §4.3 · p. 14

4.4 Output validity and response dispersion are distinct from indicator quality

  • Near-zero repeated-call dispersion and 99.98% scoreability did not establish correctness; validity, error, dispersion, and estimated cost are separate reporting dimensions.The approximately elevenfold cost difference reflects recorded provider prices and token accounting, not intrinsic computational efficiency or environmental impact.Source: §4.4 · p. 14

4.5 Implications for ecological-indicator monitoring and management

  • The proposed quality-control sequence matches checks to the target: independent measurements for concentration, deterministic unit-consistent calculations for stock, and reference separation plus order reversal for rankings.These safeguards follow observed failure modes but were not evaluated as a deployed management system; ecological interpretation also requires complementary indicators and appropriate reference systems.Source: §4.5 · p. 14

4.6 Scope, limitations and future validation

  • The benchmark covers sampled 0–10 cm areas, 30 synthetic stock scenarios, and reused paddocks; its repeated-call intervals do not estimate performance uncertainty for new farms or scenarios.It used one prompt and reasoning setting per configuration over a short period, and did not evaluate ecological thresholds, intervention outcomes, decision losses, or tool-augmented deployment.Source: §4.6 · p. 15

5 Conclusions

Across three SOC tasks, LLM reliability depended on supplied environmental information and problem representation. The benchmark separates errors in concentration, stock, and site ordering, supporting task-specific validation rather than unverified ecological use.

5 Conclusions

  • Across concentration estimation, stock calculation, and pairwise comparison, reliability depended on both supplied environmental information and problem representation.The study found residual high-SOC underprediction and local concentration errors, unit-sensitive stock calculations, and weaker comparisons for similar paddocks that were sometimes order-sensitive.Source: §5 · p. 15
  • Near-universal parseability and low repeated-call dispersion did not guarantee correct SOC outputs.The conclusions distinguish model reliability from evidence of soil-health improvement or carbon sequestration.Source: §5 · p. 15
  • Applied SOC estimates require measured validation, stock figures require auditable unit handling, and rankings require separation- and uncertainty-aware review.The benchmark is a direct-response reference for future field-oriented, tool-augmented evaluation, not evidence of management effects.Source: §5 · p. 15

Data availability and reproducibility

The source dataset is publicly available, and the manuscript documents benchmark materials and collection settings.

Data availability and reproducibility

CRediT author contributions

The contribution statement assigns writing, visualization, software, methodology, formal analysis, conceptualization, and supervision across the named authors.

CRediT author contributions

Supplementary Information Reliability of large language models for soil organic carbon indicators: An Australian benchmark of estimation, stock

The supplement documents the data, prompts, scoring, and repeated-call analyses for three SOC tasks, including position-specific pair accuracy and unit/coarse-fragment contrasts. It also reports estimated API costs and clarifies which benchmark materials are public and which remain restricted, with further release subject to author approval and privacy review.

M1 Data provenance and reference construction

  • The T1 reference set comprised 1,213 retained 0–10 cm records from 308 paddocks, with each paddock represented by its unweighted mean SOC concentration.The source documentation did not establish each record’s exact analytical method or fine-earth versus whole-soil mass denominator.Source: M1 Data provenance and reference… · p. 20

M2 Model requests and collection protocol

  • Seven configurations made independent, stateless, closed-book calls, with 20 planned responses per configuration and item or pair across the three tasks.The planned schedule comprised 616 T1 items, 120 T2 items, and 300 T3 pairs.Source: M2 Model requests and collection… · p. 20

M3 T1 prompts and supervised baselines

  • T1a added six soil-property fields to T1b’s available location, depth, and region information; label-trained baselines were contextual rather than like-for-like closed-book comparisons.The baselines included fold-specific medians and random forests using T1a or T1b inputs.Source: M3 T1 prompts and supervised baselines · pp. 20–21

M7 Post-collection descriptive displays

M8 Token accounting and estimated cost

  • Estimated API costs used normalized provider-usage accounting and recorded 29 September 2026 rates; they are standardized estimates, not invoices or comparable measures of internal computation.Source: M8 Token accounting and estimated cost · p. 23

M6 Parsing, aggregation and repeated-call uncertainty

M4 T2 scenarios, unit variants and scoring

  • For Mistral Medium 3.5, the T2 F1 − F0 contrast was -39.25 (95% interval -40.79 to -37.71), while its U×F contrast was -3.83 (-6.91 to -0.76).The 120 matched scenarios varied unit representation and coarse fragments; references used the displayed values and the specified stock convention.Source: M4 T2 scenarios, unit variants and… · p. 21, Supplementary Tables · p. 29

M5 T3 pairs and secondary model

  • The evaluation sampled 300 unordered paddock pairs without replacement from 47,278 eligible pairs, without stratification by SOC separation (GAP), and described each paddock using its available T1a fields.For each configuration and pair, ten presentations placed the higher-reference-SOC paddock at A and ten at B.Source: M5 T3 pairs and secondary model · p. 22

Data and code availability

  • SOC-M v1, the 120 T2 scenarios and displayed-value references are publicly available, while other benchmark materials remain in the Project 2 archive.FarmData coordinates, authentication credentials and raw response ledgers are excluded; further release requires author approval and privacy review.Source: Data and code availability · p. 35

Related papers

Loading 2610.18201v1…