Source-linked AI summary
EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards
Zhiqing Cui, Xinxiang Yin, Yihong Tang, Xinglang Zhang, Yuanzhe Hu, Siru Zhong, Weidong Tang, Yuxuan Liang, Weijia Li, Ming Jin, Shirui Pan, Yuhao Kang, Dingyi Zhuang, Jinhua Zhao
TL;DR
EarthVerse addresses the gap between interpreting supplied Earth-science observations and constructing a compatible, traceable evidence base for hazard investigations. It benchmarks agents on reproducible package-scoped tasks with executable answer units and process rubrics under controlled evaluation. Across systems, high local accuracy does not translate into reliable end-to-end completion.
Problem
Existing benchmarks usually provide the relevant observation or evidence layer in advance, leaving open whether agents can select compatible sources, align them, and preserve support across an investigation.
Method
EarthVerse evaluates 405 reproducible multi-source investigations using executable answer units, task-specific process rubrics, and multiple valid research paths.
Results
84.65% is the best mean answer-unit accuracy across 25 systems, while the highest Strict@95 is 34.81%.
Takeaways & Limitations
Reliable scientific agents must maintain editable links among claims, sources, scales, calculations, and physical interpretation throughout the investigation.
Takeaways & Limitations
The benchmark is harder than settings where passages, figures, or named observations are selected and the required output is already clear.
Abstract
from arXiv · showhide
Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism. We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations. Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families. Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer. We provide executable ground truth that decomposes each task into fine-grained answer units, together with task-specific rubrics that assess the supporting research process while allowing multiple valid paths. We evaluate 25 model and agent systems under a controlled tool-using protocol, then use controlled studies to locate failures in evidence access, tool selection, memory, reasoning, interaction, and scientific execution. Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%. The gap shows that current agents often complete individual steps without maintaining a consistent chain across evidence, scales, units, calculations, and physical interpretation. EarthVerse provides a reproducible basis for measuring end-to-end scientific reliability in dynamic Earth systems.
1 Introduction
EarthVerse addresses the difficulty of reconstructing hazard processes from heterogeneous observations while preserving traceable claim–evidence links. Its results show a substantial gap between local accuracy and reliable end-to-end completion.
- Motivation: Natural-hazard analysis must reconcile observations collected for different purposes, variables, spatial footprints, and time windows before calculation.This reconciliation affects estimates of severity, exposure, mechanism, and response.
- Research gap: Existing Earth-science benchmarks usually supply the relevant passage, figure, image, or observation layer rather than requiring agents to construct an evidence base.EarthVerse targets source selection, cross-source calculation, and provenance preservation as part of the investigation.
- Benchmark focus: EarthVerse evaluates whether heterogeneous observations remain aligned in scale, variable meaning, time window, units, and transformations throughout an evolving claim–evidence state.A broken binding can make a locally correct calculation describe the wrong Earth process.
- Benchmark scope: 405 investigations grounded in 199 real disasters and extreme events require evidence from more than seven distinct sources on average.Each event package contains about 34 files from 80 external source and access families, making source choice a scientific judgment.
- Headline result: 84.65% is the best mean answer-unit accuracy across 25 systems, compared with only 34.81% for the highest Strict@95.The gap indicates that systems can recover many local facts and calculations while still missing a source, scale match, or mechanism needed for a reliable account.
- Contributions: EarthVerse contributes executable answer units, task-specific process rubrics, and controlled studies of evidence access, tools, memory, reasoning, interaction, and execution.The benchmark allows multiple valid investigative paths while assessing whether the resulting evidence and calculations support the conclusion.
2 Related Work
Prior Earth-science and disaster benchmarks increasingly add multimodality and tools, but generally begin with observations or layers selected in advance. EarthVerse combines flexible scientific tooling with the open-ended task of selecting compatible evidence.
- Earth observation: Most Earth-science benchmarks isolate figure interpretation, atmospheric processes, geospatial relations, remote sensing, or forecasting over prepared observations.Their primary strength is controlled perception or interpretation rather than evidence construction.
- Disaster intelligence: Disaster benchmarks have progressed from classification and damage mapping toward tool-supported emergency analysis, while retaining declared observations or layers.DORA is cited as a closer operational workflow with expert-authored tasks, typed tools, and replayable reference calls.
- Scientific agents: Scientific-agent research introduced reasoning–action loops, executable environments, code and data workflows, and long-horizon orchestration.BLADE and DiscoveryBench are relevant because they permit multiple valid analysis paths while keeping work evaluable.
- EarthVerse position: EarthVerse couples expert authoring with numerical, atmospheric, and geospatial tools, scoring source choice, intermediate work, and physical interpretation separately from the final answer.The tool suite supports executable analysis while leaving the scientific strategy open.
- EarthVerse position: EarthVerse evaluates whether agents can choose compatible evidence, use scientific tools appropriately, and maintain a coherent account as an event analysis develops.This combines retrieval, tool use, and scientific synthesis rather than treating them as separate capabilities.
3 The EarthVerse Benchmark
EarthVerse packages real hazard investigations as reproducible, multi-source tasks in which agents discover evidence, perform transparent calculations, revise interpretations, and document scientific support. Fine-grained answer units and expert review make diverse investigative paths checkable.
- Benchmark scope: 405 investigations cover 199 documented disasters and extreme events, 19 hazard families, and 6,709 local files.Tasks follow post-event inquiry from establishing what happened through evidence selection, mechanism testing, reconciliation, and supported conclusions.
- Scientific challenge: EarthVerse treats misalignment across windows, footprints, variables, and units as part of the scientific problem.Transparent operations include aggregation, normalization, ratios, thresholds, lags, spatial overlap, weighted indices, and bounded counterfactuals.
- Investigation workflow: A solver alternates targeted evidence access, calculation, and physical interpretation, revising the event picture when observations conflict with the current account.Report-first and time-series-first trajectories can both receive full credit when they establish the required quantities and mechanism.
- Evidence discovery: Each released task requires 7.26 distinct sources on average selected from a package with a median of 34 candidates.Prompts name the event and scientific decision but do not identify relevant files or reveal an answer path.
- Scientific checkability: Conclusions are scored with supporting values, units, source roles, and mechanism tests, attaching each operation to the evidence it uses.The benchmark tests scientific checkability rather than recall of undocumented formulas.
- Task construction: Every task includes an expert solution, structured ground truth, a deterministic compute_gt.py program, and a task-specific process rubric.The release contains 10,879 answer units, each representing a nontrivial scientific obligation such as a window, ranking, quantity, mechanism, or qualification.
- Research environment: The shared research environment provides six general operations: file discovery, source reading, local text search, structured-data inspection, scoped Python execution, and answer finalization.The registry also supports custom registration and retains reusable operations for reports, time series, vector, exposure, raster, and Earth-observation analysis.
- Quality assurance: Human review checks event identity, source reliability, temporal and spatial fit, solutions, ground truth, executable programs, labels, and rubrics before release.Package-local data must regenerate every deterministic value, and the visible assignment must match the hidden scoring target.
4 Evaluation Protocol
The evaluation protocol combines endpoint correctness with the quality of the documented research process under a common package interface. It measures partial completion and near-complete reliability while allowing multiple defensible investigative paths.
- Protocol: EarthVerse fixes the scientific question, evidence boundary, execution budget, and answer contract while agents select evidence, calculate, record provenance, and submit structured answers.The same interface supports controlled comparison across systems and modalities.
- Outcome scoring: Answer correctness is Ai = min(Hi, Ui), so a fluent answer cannot exceed the scientific obligations it actually completes.Ui is required answer-unit completion and Hi is holistic answer judgment, both on a 0–100 scale.
- Core metric: The primary score gives equal weight to endpoint correctness and the documented research process.It combines answer correctness with the task-specific process-rubric score Pi.
- Answer units: Fine-grained answer units expose which obligations failed instead of allowing a holistic grade to conceal missing scientific work.Tasks can combine windows, quantities, comparisons, mechanisms, rankings, and open-ended conclusions.
- Reliability metrics: Mean unit accuracy captures partial completion, whereas Strict@95 measures the percentage of tasks with Ui ≥95 and therefore near-complete reliability.Numerical answer units use task-specific tolerances; trajectory statistics do not enter the score.
- Path pluralism: EarthVerse does not prescribe a tool order because complex multi-source investigations may support multiple defensible trajectories.Credit depends on what the system establishes and whether evidence and calculations support it, not on reproducing a reference path.
5 Main Evaluation
EarthVerse evaluates whether scientific agents can maintain a valid, auditable evidence chain across package-scoped hazard investigations. Results show that high local accuracy, extended interaction, and stronger domain recognition do not guarantee reliable end-to-end completion.
- Overall system performance: 84.65% best mean answer-unit accuracy contrasts with only 34.81% highest Strict@95, exposing a gap between local competence and conjunctive reliability.A single missing source, window, unit, calculation, or mechanism can invalidate an otherwise plausible account.
- Overall system performance: Useful interaction changes the claim–evidence state, whereas longer traces that preserve weak sources or stale values add cost without improving quality.OpenResearcher uses 39.73 rounds yet trails Claude Fable 5 by 54.18 Core points, while GeoMMAgent’s five rounds leave evidence undiscovered.
- Overall system performance: Domain-specialized open models remain 40–70 Core points behind frontier general systems, with the gap appearing in source selection, provenance, revision, and stopping.Earth-specialized pretraining improves terminology and process recognition but has not yet produced a reliable research loop in the evaluated systems.
- Reasoning, interaction, and evidence control: Reasoning helps when evidence is editable: interactive retrieval raises Core from 51.30 to 73.80, and final review raises xhigh from 68.58 to 86.29.With fixed evidence, additional effort can reinforce an early error; review can also add cost once the claim–evidence state is coherent.
- Reasoning, interaction, and evidence control: Evidence localization is the clearest reproducible bottleneck, raising Core by 14.72 points and improving spatiotemporal, mechanism, and calculation capabilities.Relevant-file localization lifts spatiotemporal Core from 64.40 to 91.47, physical-mechanism Core from 65.15 to 80.61, and quantitative-calculation Core from 73.43 to 87.57.
- Cross-benchmark transfer and benchmark adaptation: Supplied-observation benchmarks can show high adapted performance, but EarthVerse Strict@95 remains below 35% because agents must construct, check, and revise the evidence base.EarthSE fill-in accuracy rises from 20.29 to 61.90 and GeoMMBench accuracy from 84.52 to 98.01 after interface and answer-structure corrections.
6 Conclusion
EarthVerse shows that scientific-agent reliability depends on preserving a complete, editable claim–evidence state rather than accumulating locally correct steps. Its controlled studies identify evidence access, routing, memory, interaction, and execution controls that shape whether analyses remain scientifically coherent.
- Conclusion: Scientific reliability depends on preserving links among claims, sources, scales, and calculations throughout an investigation.A single unsupported source, scale mismatch, or missing mechanism can change the result.
- Conclusion: Reasoning helps when interactive controls let agents replace weak sources, recompute quantities, and carry corrections into the final account.The controller must preserve claim–source bindings, check method applicability, and stop when required evidence is complete.
- Conclusion: Interpreting supplied evidence is easier than constructing and maintaining a compatible evidence base.EarthVerse is harder because agents must decide what belongs in the analysis and preserve compatibility through calculation and synthesis.
- Conclusion: EarthVerse contributes an executable benchmark and controlled diagnostics for multi-source hazard research.The benchmark traces failures across evidence, computation, and reporting while allowing multiple valid research paths.
- Controlled studies: The full catalog loses 9.80 Core points relative to the six-tool foundation while using 19% more tokens.Additional tools can introduce incompatible assumptions about variables, units, aggregation, and applicability.
- Controlled studies: Top-20 routing recovers 11.21 Core points over the full catalog with 34% fewer tokens.Routing works best as a revisable prior that preserves a general core and introduces specialist methods when required variables appear.
- Controlled studies: Long bounded retention gains 13.66 Core points over aggressive compression and 5.27 over full replay.Useful scientific memory is a compact ledger of active claims, sources, units, and unresolved conflicts rather than a verbatim transcript.
- Controlled studies: Additional interaction helps when it closes a named evidence gap and hurts when it reopens settled calculations or follows an unproductive branch.Stopping should depend on evidence coverage, provenance checks, unit checks, and resolved or disclosed contradictions.
A.4 Professional meteorological environments
EarthVerse tests whether scientific environments support valid methods under package-specific data constraints, not merely whether tool calls succeed. The results favor tools whose operations match available structures and expose when required variables or sampling support are absent.
- A.4 Professional meteorological environments: EarthVerse requires repeated use of one of 13 libraries or runtimes under a common protocol to separate scientific fit from technically successful misuse.Adapters expose implemented operations and record whether calls succeed.
- A.4 Professional meteorological environments: Julia gives the largest gain (+4.62), while Julia, R climate, MetPy, xclim, and CDO improve Core when their operations match package structures.These tools compress low-level numerical steps into inspectable transformations without hiding variables or aggregation.
- A.4 Professional meteorological environments: Required-use settings expose failures that optional-tool studies can miss when event packages lack suitable time series or paired observations.Under hard requirements, agents may bend available data toward an unsuitable method instead of rejecting it.
- A.4 Professional meteorological environments: Execution success is not scientific validity: specialist methods help only when their required variables and sampling support exist.Otherwise, a successful call can produce a precise result that the package cannot justify.
- A.4 Professional meteorological environments: A complete research loop requires agents to select evidence, transform it, revise event-scale explanations, and retain replayable provenance.This begins earlier than supplied-observation question answering and demands more from the controller than tool-execution benchmarks.
- A.4 Professional meteorological environments: Strong systems often interpret selected observations correctly but lose reliability when choosing records, scales, compatible variables, and completion points.Many failures originate when plausible values are attached to the wrong aggregation, time window, unit, or source role.
- A.4 Professional meteorological environments: Scientific tools should expose applicability conditions, while justified refusal should receive credit when inputs are inadequate.Record length, sampling support, missing-data rules, and estimator assumptions determine whether a computed value supports the claim.
B.4 Limitations
The paper bounds EarthVerse as a controlled, reproducible approximation of scientific research with known coverage, judging, provenance, and statistical-scope limitations. These constraints shape how benchmark results should be interpreted and extended.
- B.4 Limitations: EarthVerse does not cover the full range of Earth-system science, long-term attribution, or live forecasting.Package isolation improves reproducibility and blocks web leakage, but limits open-ended discovery and institutional data-quality judgments.
- B.4 Limitations: Masked products prevent tests of absolute geolocation.This limits what can be inferred about visual and spatial capabilities from the benchmark.
- B.4 Limitations: Structured ground truth chooses one decomposition of a scientific answer, while process scoring relies on an LLM judge that can err.Future releases should measure agreement with domain experts on a stratified subset.
- B.4 Limitations: Some historical events and source documents may have appeared in pretraining despite package isolation and hidden scoring artifacts.Package-specific calculations make shortcut answering harder but do not eliminate possible prior exposure.
- B.4 Limitations: Controlled studies use paired subsets, and smaller changes need replication across more packages and backbones.Cross-benchmark revision measures recoverability under feedback rather than zero-shot superiority.
C.2 Shared experimental configuration
EarthVerse standardizes package-scoped investigations with hidden evidence boundaries, explicit harness controls, and diverse source and access families. Its shared configuration supports reproducible analysis while preserving event-specific source selection and cross-system diagnostic comparisons.
- Execution boundary: Every run restricts reads, writes, and Python execution to the active evidence collection while returning local text, structured summaries, metadata, and measurements.The solution, rubric, ground truth, scoring code, prior outputs, and web evidence remain hidden.
- Harnesses: The standard EarthVerse harness controls model rows, while selected systems use model-native harnesses or their own controllers.Table 2 identifies these harness differences explicitly.
- Resource controls: The common limit is 48 turns, 1,680 seconds for research, and 1,800 seconds overall, with bounded context, tool observations, Python calls, repairs, and finalization.The controller retains 16 recent messages and a 24,000-character research state; tool observations are capped at 12,000 characters.
- Package sources: EarthVerse consolidates 6,397 provenance records and 6,709 local files into 80 external source and access families across 199 event packages.Aliases are grouped at the product/API-family level while distinct providers, sensors, versions, and access semantics remain separate.
- Source coverage: The packages span event discovery, authoritative reports, meteorology, climate, hydrology, ocean, air quality, remote sensing, geospatial, humanitarian, and impact sources.Examples include NASA, NOAA, WMO, UN agencies, national meteorological services, CHIRPS, GPM IMERG, ERA5-Land, Sentinel-1, and humanitarian APIs.
- Shared diagnostics: The strongest systems lead across nearly every diagnostic tag, but residual errors concentrate in connections between results and compatible evidence.Among the top five, causal-chain scores trail quantitative calculation by 4.9–7.9 points; GeoMMAgent reaches 58.82 on remote sensing after often bypassing raster intermediates.
D.2 Preference structure and diagnostic error paths
Preference and trajectory diagnostics show that leaderboard strength does not guarantee complete investigations. Errors become consequential when systems fail to revise, recover, or preserve connections across evidence, calculations, and final synthesis.
- Preference structure: Pairwise judgments broadly recover leaderboard order while revealing uncertainty hidden by Core alone.Leaders win consistently, adjacent frontier systems remain close, and ties are uncommon.
- Preference structure: The drop beyond the leading group aligns with low Strict@95, indicating that partial success is common while consistently complete research remains rare.Figure 7 separates wins, losses, and ties in opponent-balanced comparisons.
- Task-level diagnostics: Frontier models retain a dense high-score mode and long tail toward zero, whereas agent frameworks cluster moderately high and open-weight or Earth-specialized systems shift downward.Error counts overlap more than Core scores, so strong systems’ advantage lies partly in catching or containing mistakes.
- Error paths: Frontier systems more often keep affected tasks in the 40–70 or above-70 bands, while open-weight and Earth-specialized systems more often fall below 40.Recovery, cross-source checking, and final synthesis determine whether local mistakes spread through the answer.
- Error paths: The largest within-system Core losses accompany premature stopping, failed recovery, tool failure, and missing required outputs.Numerical and schema errors are common but often have smaller average effects because some runs absorb them before finalization.
- Trajectory diagnostics: Higher reasoning effort changes evidence treatment more than collection volume: xhigh opens about as many files as medium effort, uses fewer rounds, and takes almost twice as long.Additional work occurs between observations through interpretation comparison, calculation review, and completeness decisions.
- Direct conditions: Fixed direct serialization prevents models from returning to sources after discovering an incorrect window, unit, or variable.Compressed direct includes more files in far fewer tokens than all-text, but neither mode supports follow-up measurement.
D.3.3 Evaluation cost
EarthVerse separates evaluation quality from resource expenditure and tests how evidence access, retention, catalogs, and specialist execution affect performance. The results show that cost, context, and tool availability create distinct continuity trade-offs.
- Evaluation cost: Quality and cost do not form a simple ladder across evaluated runs.The strongest frontier evaluations exceed $2,500, while low-price systems can have substantially different Core scores.
- Evaluation cost: GPT-5.6 Terra retains about 98% of GPT-5.6 Sol’s Core at 44% of its cost.The two Flash systems remain below $45 but differ by almost 14 Core points.
- Trajectory effects: Aggressive compression produces more file listings and evidence references but the lowest process score because the agent rebuilds discarded context.The expert-only suite is efficient for the opposite reason: it achieves high call success while inspecting the fewest distinct files.
- Intervention effects: With 170 visible tools, multi-source performance falls to 47.50, while aggressive compression loses 20.07 causal-chain points relative to long retention.A large catalog destabilizes the next action; compression destabilizes the meaning of earlier actions.
- Evidence localization: Only evidence-localization oracles have intervals excluding zero: relevant files [+5.48, +24.69] and the evidence map [+2.34, +21.25].The stable gain comes from knowing where event-specific evidence is, rather than receiving a generic plan.
- Capability effects: Both effective oracles exceed 90 on spatiotemporal reconstruction compared with 64.40 at baseline, while improving ranking and decision less.Localization resolves which records contain the relevant event window and region but does not determine how competing quantities should be weighted.
D.6.1 Compliance, execution reliability, and implemented scope
The professional-environment study distinguishes willingness to invoke tools from successful execution and scientific applicability. EarthVerse therefore evaluates not only whether code runs, but whether operations fit the available records and support the final scientific claim.
- Compliance and execution: Nearly every run attempts the required specialist interaction, so tool-call willingness is not the bottleneck; execution is.Specialist-call success ranges from 20.29% for PyET to 97.56% for R climate.
- Implemented scope: R climate, Julia, and CDO often accept available time series or grids directly, whereas other adapters require records or variables missing from many packages.The ranking reflects overlap between audited operations and package data, not a comparison of programming languages.
- Applicability and reproducibility: A successful call proves only that code ran; it does not establish estimator fit, adequate sampling geometry, or an answer to the scientific question.These checks occur between execution and scientific use, and execution reliability cannot be collapsed into final quality.
- Applicability and reproducibility: Adapters record normalized arguments, status, outputs, and errors while validating package-scoped inputs before execution.This makes rejected operations reproducible by showing why a record was too short or a required variable was absent.
- Execution contract: The solver operates under strict package-only visibility and must inspect local files, execute tools, and produce structured evidence before finalizing.Ground truth and reference materials remain hidden from the solver.
- Execution contract: Event validation requires traceable confirmation of occurrence, source reliability, and correspondence in name, time, place, and hazard type.Government, international, peer-reviewed, and authoritative scientific sources are preferred over encyclopedias and generic search pages.
- Illustrative failure: One hosted GPT-4o run scored Answer 40.00 and Rubric 42.50 after identifying the mechanism but reporting the wrong required ratio.The trace contained 19 model rounds and nine tool calls, including two guessed-path failures.
COMPUTED LEDGER
The computed ledger tests hazard mechanisms by combining heterogeneous observations, explicit thresholds, and reproducible calculations. Across cases, it preserves source compatibility and exposes when substitutions or repairs change the scientific conclusion.
- Burn-scar ledger: The burn-scar analysis reconciles three precipitation products while separating event-window dNBR from annual embedding change.The trace also anchors two report claims and preserves a failed semantic search followed by filename-based recovery.
- NYC Ida: The NYC Ida event combines an 88.138 mm record hour with a 254.000 mm regional reference and a 136.888 mm grid peak.The ledger also reports 5.013 million exposed people and a 267.812 million-person-mm rain load.
- NYC Ida: The NYC Ida baseline response-stress index is 72.6 and rises to 80.1 under the prescribed future perturbation (+7.5).The local hourly record remains distinct from the wider gridded rain shield.
- Heat-source audit: The ERA5-Land aggregate substitution fails because it lacks apparent temperature and daily minimum temperature, preventing the requested heat-duration and warm-night metrics.Its maximum is 35.3 C, 9.3 C below the local peak, and the corrected label is sustained_apparent_heat_with_warm_night_exposure.
- Sahel heat: The Sahel workflow verifies a 0.3272 one-hour share, a 0.963 annual ratio, and four true threshold flags for the coordinated hazard ledger.The ledger combines heat, wet-bulb temperature, exposure, infrastructure, gridded context, and surface stability, while retaining an explicit failed computation and repair.
INDICES AND +1 C SCENARIO
The scenario and failure cases show that computed indices can quantify increased response burden, while correct mechanism identification does not guarantee the requested physical measure. Reliable completion therefore depends on matching the calculation to the scientific target and retaining supporting evidence.
- Sahel scenario: The Sahel +1 C scenario raises heat response from 77.344912 to 84.444912 and priority from 86.374648 to 93.094648.The reported deltas are +7.1 for heat response and +6.72 for priority.
- Kilauea failure: The Kilauea case identifies coupled summit drainage and lower-rift effusion but returns lava effusion rate 100 m3/s instead of the required volume ratio.The run finalizes without computing the 0.8/0.8 volume ratio.
- Kilauea failure: The required summit-collapse-to-lava-flow volume ratio is 1.0, based on approximately 0.8 km3 for each volume.A complete answer also requires summit evidence and rejection of isolated-rift, isolated-summit, and rainfall-triggered alternatives.
- Kilauea failure: The Kilauea error is scientific and evidential rather than formatting-related: the mechanism is correct, but the selected index is irrelevant to the requested target.The model also omits summit evidence and explicit alternative-mechanism rejection.