PaperlayerView exact revision on arXiv

Source-linked AI summary

Google Earth AI Geospatial Reasoning Agent: Capabilities and a Systematic Evaluation

Noam Slonim, Aviv Slobodkin, Gil Fidel, Gal Moshitch, Mimi Sun, Lily Mihalkova, Joydeep Paul, Per Bjornsson, Sydney Lin, Jamie McPike, Samapriya Roy, Utkarsh Patel, Gia Jung, Zachary Luna, Allison Berman, William Cai, Xiao Wu, Mohit Agarwal, Mandar Sharma, Aliza Hoffman, Mark Young, Rama Pasumarthi, Roy Lee, Ruth Alcantara, Luke Barrington, Aaron Bell, Reuven Sayag, Avinatan Hassidim, Yossi Matias, Tyler McHenry, Ronald Ho, Christopher Haire, Oliver Guinan, Niv Efron, Tim Thelin, Gautam Prasad, Shravya Shetty

arXiv:2610.15750v1cs.AI

TL;DR

Evaluating geospatial agents is difficult because real-world problems allow multiple workflows, require discovery across large catalogs, and depend on changing ground truth. The paper introduces benchmarks and compares four Gemini-based configurations to separate the contributions of tools and orchestration. GRA performs well across open-ended and templated tasks, with geospatial tools providing the largest gain and planning adding further improvement on open-ended tasks.

  • Problem

    Geospatial-agent evaluation must handle multiple valid workflows, data discovery across heterogeneous catalogs, and ground truth that changes over time.

    Source: §1 · p. 2

  • Method

    The authors introduce GeoExpertQueries and GeoKTools and compare four configurations of the same Gemini model across those and two external benchmarks.

    Source: §1 · p. 3

  • Results

    GRA substantially outperforms simpler baselines on GeoExpertQueries, GeoKTools, and CBGB; its geospatial skills and tools drive the largest gain, with planning adding further improvement on open-ended problems.

    Source: §1 · p. 3, §6 · p. 25

  • Takeaways & Limitations

    UnivEARTH has become saturated, while geospatial benchmarks such as CBGB can also drift as underlying imagery and boundaries change.

    Source: §6 · p. 25

    Real-world enterprise queries remain unevaluated, and such evaluation faces data availability, sensitivity, and reproducibility challenges.

    Source: §6.2 · p. 26

Abstract

from arXiv · show

Google Earth AI is a family of geospatial AI models, together with an agent that orchestrates them. We present this agent, Google Earth AI's Geospatial Reasoning agent (GRA), a Gemini-powered research system that combines geospatial data sources, foundation models, and tools to answer questions about our planet. GRA organizes these into modular domain skills and pairs them with upfront geospatial planning to decompose complex queries into multi-stage workflows. Evaluating such systems is hard: real-world problems admit multiple valid workflows, require data discovery across large catalogs, and rely on evolving ground truth. We introduce two benchmarks and release their development splits. GeoExpertQueries poses complex analytical problems grounded in practitioner use cases and refined by geospatial experts; each is paired with a consensus golden workflow from three independent experts and scored by a human-validated, tool-agnostic autorater. GeoKTools consists of templated queries with deterministic reference solvers that systematically vary the number of required tool types. We also use two external benchmarks, CBGB and UnivEARTH; the latter we find to be saturated. To isolate the contribution of GRA's tools and its agentic orchestration, we compare four configurations, each with Gemini 3.1 Pro and 3.8 Flash: a standalone LLM, the LLM with basic tools, the LLM with GRA's skills and tools but without its upfront planning and execution guidance, and GRA itself. With at most basic tools, an advanced model recovers much of the answer on templated queries but satisfies only a small fraction of the expert workflows on open-ended problems; GRA performs well on both. The largest step comes from GRA's skills and tools; on open-ended problems, its planning and execution guidance add a further gain with Gemini 3.8 Flash, letting Flash match or exceed Pro at lower cost.

1. Introduction

Real-world geospatial analysis requires integrating heterogeneous data, but evaluating agents is difficult because workflows vary, data must be discovered, and ground truth changes. This study introduces benchmarks and controlled comparisons to measure the contributions of geospatial tools and orchestration.

Evaluation design

  • The four configurations hold the Gemini model fixed while adding basic tools, GRA’s geospatial skills and tools, and then planning and execution guidance.This comparison separates the contributions of domain capabilities from those of upfront planning and execution guidance.Source: §1 · p. 3

Benchmarks

  • GeoExpertQueries pairs practitioner-grounded open-ended problems with consensus workflows from three experts and human-validated, tool-agnostic scoring.The benchmark has 101 problems, with 76 held out for testing.Source: §1 · p. 3
  • GeoKTools uses deterministic reference solvers and varies the number of required tool types across templated queries.It includes 305 queries from 92 expert-written templates spanning 24 tool types, with K=1, 2, 3, or 4.Source: §1 · p. 3

Key findings

  • GRA outperforms simpler baselines on GeoExpertQueries, GeoKTools, and CBGB; on GeoKTools it scores 86 versus 41 for the strongest baseline.With Gemini 3.8 Flash, GRA’s skills and tools raise GeoExpertQueries performance from 3 to 52, and planning and execution guidance raise it from 52 to 63.Source: §1 · pp. 3–4
  • UnivEARTH is saturated: every configuration scores above 85, including a standalone LLM without Earth Engine access.The authors attribute this to models answering from published source text without geospatial analysis.Source: §1 · p. 4

2. Related Work

Prior geospatial-agent research spans specialized tools, agent architectures, and benchmark paradigms, but the cited benchmarks generally isolate capabilities or constrain workflows. The paper positions its evaluation suite as addressing the gap in end-to-end, open-ended workflow evaluation.

Geospatial agents

  • Prior academic systems such as ClimateAgent (Li et al., 2025b), NORA (Zhou et al., 2026), and Earth-Agent pair LLMs with specialized geospatial tools.GRA additionally integrates Google foundation models, forecasting systems, Google Maps Data, and other proprietary assets.Source: §2 · pp. 4–5

Agent design

  • GRA combines on-demand skills with planning before tool use, designs proposed in other agent work.The paper evaluates this combined layer against a configuration with the same model, skills, and tools but without it.Source: §2 · p. 5

Geospatial benchmarks

  • Existing benchmarks often test code generation, tool calling, imagery processing, or planning separately rather than end-to-end multi-source geospatial workflows.The paper identifies no prior benchmark combining open-ended practitioner tasks, independent expert workflows, tool-agnostic scoring, and fixed-model configuration comparisons.Source: §2 · p. 5

This evaluation

  • GeoExpertQueries scores open-ended practitioner problems against independently authored expert workflows, while GeoKTools varies the known number of required tool types.CBGB and UnivEARTH provide external checks and are adapted so systems with different tool sets can be compared.Source: §2 · pp. 5–6

3. Evaluated Systems

The evaluated systems use the same Gemini model within four configurations, separating general tools, GRA’s geospatial capabilities, and its orchestration. GRA combines specialized geospatial tools with on-demand skills, upfront planning, and adaptive execution guidance.

3.1. The Geospatial Reasoning Agent

  • GRA’s domain tools cover satellite and geospatial data, demographics, maps, boundaries, environmental prediction, remote-sensing search, climate risk, time series, and spatial prediction.Across evaluated runs, it loaded 23 skills and called 62 distinct domain and workspace tools.Source: §3.1 · p. 7
  • On-demand skills and external scratch datasets keep unused tool definitions and large spatial intermediates out of the model context.The agent plans retrieval and processing before tool use, then may adapt the plan as outputs reveal a more direct route or data gap.Source: §3.1 · pp. 8–9

3.2. Evaluated Configurations

  • The four configurations progress from a standalone LLM to basic tools, GRA’s skills and tools in a reactive loop, and full GRA with planning and execution guidance.All four use the same underlying Gemini model.Source: §3.2 · pp. 9–10

3.3. Models

  • The comparison uses Gemini 3.1 Pro and Gemini 3.8 Flash, which differ in capacity tier, release generation, and default thinking level.The authors therefore describe differences between models as reflecting the combined effect of those factors.Source: §3.3 · p. 10

3.4. Experimental Setup

  • Best-effort mode asks GRA to use the closest viable substitute when ideal data is unavailable and report substitutions and uncertainty.Other configurations receive the same instruction not to decline, but not GRA’s substitution instructions.Source: §3.4 · p. 10
  • Every query runs three times per configuration and model, with benchmark scores reported as the mean and standard deviation across runs.Runs are limited to 100 LLM calls and 30 minutes; exceeding either limit or the context window results in a zero score.Source: §3.4 · pp. 10–11

4. Benchmarks

The section introduces four complementary benchmarks for open-ended geospatial analysis, controlled multi-tool tasks, and external evaluation. It describes how they are constructed and scored, and reports that UnivEARTH is saturated.

4. Benchmarks

  • GeoExpertQueries tests open-ended practitioner problems, while GeoKTools uses deterministic templated queries; both internal benchmarks’ development splits are released.CBGB and UnivEARTH provide external checks on the internally developed benchmarks.Source: §4 · p. 11

4.4.2. UnivEARTH

  • A standalone LLM scores 87 to 94 on UnivEARTH, which the authors treat as benchmark saturation rather than a capability measurement.The benchmark’s discriminative power can decline once public items are accessible through model training data.Source: §4 · p. 11

4.1.1. Construction

  • Three independent experts created workflows for practitioner-derived queries; incompatible approaches remained as alternatives for four queries, and filtering unsolvable items left 101 of 132.The 101 queries cover four domains and were randomly divided within each domain into 25 development and 76 test queries; the smallest domain has only four test queries.Source: §4.1.1 · p. 12, §4.1.2 · p. 13

4.1.3. Evaluation

  • The tool-agnostic autorater scores executed analysis against expert workflows, and its agreement with experts is comparable to inter-expert agreement (Spearman correlation 0.72).It accepts equivalent methods and tools, while withholding credit for merely proposed work, fabricated outputs, or analysis not performed on retrieved data.Source: §4.1.3 · pp. 13–14, §4.1.4 · p. 14

4.2. GeoKTools

  • GeoKTools varies required tool types from K=1 to K=4 and uses reference solvers to generate verifiable answers; its 305 queries come from 92 retained templates.The 220-query test split is separated from development at the template level to prevent instances from the same template crossing splits; scoring uses exact matches or an LLM judge for partial credit.Source: §4.2 · p. 15, §4.2.1 · p. 16, §4.2.2 · p. 16

4.3. Data Release

  • The released development splits include GeoExpertQueries’ queries and workflows and GeoKTools’ queries, answers, solver specifications, and scoring code.The test splits are withheld to preserve an uncontaminated evaluation signal; live-source answer values may change when solvers are rerun.Source: §4.3 · pp. 16–17

4.4. External Benchmarks

  • For external evaluation, CBGB tests numeric-answer geospatial questions, while UnivEARTH requires imagery-catalog discovery as well as analysis.Both are presented as questions rather than requests for Earth Engine code.Source: §4.4 · p. 17

4.4.1. CBGB

  • CBGB is evaluated as a geospatial question-solving task rather than code generation, so these results are not directly comparable with its original evaluation.The evaluation uses 44 questions, excludes one item whose reference solution exceeds execution limits, and updates four answers to reflect changed source data.Source: §4.4.1 · pp. 17–18

5. Results

Across three benchmarks, GRA substantially outperforms standalone and basic-tool baselines; its geospatial tools provide most of the gain, while planning further helps Gemini 3.8 Flash on open-ended queries. UnivEARTH is saturated, and Flash delivers stronger GRA scores at lower inference cost.

5.1. Main Results

  • GRA’s geospatial tools account for most gains; upfront planning adds 10.5 GeoExpertQueries points with Flash, but Pro scores 3.2 points lower than tools-only.On GeoKTools and CBGB, the two tool-equipped configurations score within three points, with no consistent planning gain.Source: §5.1 · pp. 18–19
  • GRA with Gemini 3.8 Flash scores 62.7 on GeoExpertQueries (versus 3.3 maximum), 85.9 on GeoKTools (versus 40.5), and above 80 on CBGB (baselines below 20).Table 4 summarizes the main test-set results; scores are means over three runs.Source: §5.1 · p. 18, Table 4 · p. 19

5.2. GeoExpertQueries: Expert-Level Geospatial Reasoning

  • Flash’s planning gain concentrates in Natural Hazards (+20.4 points); plans steer queries away from web lookup, while Pro’s stricter plan-following may impede expected workflow steps.The authors attribute the uneven effects to differences in how the models follow plans and adapt during execution.Source: §5.2 · p. 20

5.3. GeoKTools: Effect of Tool Complexity

  • With GRA tools, scores stay between 79.4 and 91.2 across K, while baseline scores decline as K rises; K counts tool types, not intrinsic difficulty.The K=1, 3, and 4 cells have only 6–9 templates each, so a few templates can shift a cell’s score.Source: §5.3 · p. 21, Table 6 · p. 21

5.4. CBGB: External Corroboration

  • GRA’s tool-equipped configurations score 95.6–97.8 on CBGB’s Easy tier and 77.8 on Difficult, while baselines fail almost all Difficult questions.The reported CBGB numbers are not comparable with the published protocol because this evaluation setup differs.Source: §5.4 · pp. 21–22, Table 7 · p. 22

5.5. UnivEARTH: Benchmark Saturation

  • On UnivEARTH’s 140-question test, configurations score 86.9–98.8 versus roughly 33% for the original evaluation’s best systems; standalone LLMs answer 100% without computation (90.2% correctly), motivating a benchmark-durability rather than capability interpretation.Source: Table 8 · p. 22, §5.5 · pp. 22–23

5.6. Model Comparison and Cost-Accuracy Tradeoffs

  • Gemini 3.8 Flash makes GRA 38–71% cheaper than Pro across benchmarks while scoring higher on all four.On GeoExpertQueries, Flash costs $0.199 per query against Pro’s $0.688, with a 15.1-point advantage.Source: §5.6 · p. 23

5.7. Capability and Tool Usage

  • On GeoExpertQueries with Flash, GRA reduces web searches from 7.0 to 3.2 per query and loads Earth Engine more often (54% versus 38%); spatial-compute skills dominate usage.Earth-observation skills are loaded on at most 4% of queries, partly because general-purpose Earth Engine skills are counted under spatial compute.Source: §5.7 · pp. 23–24

6. Conclusions and Future Work

The paper evaluates geospatial reasoning agents with two new benchmarks and two external benchmarks, finding consistent advantages for GRA on several benchmarks. It also identifies limits in benchmark coverage and evaluation, and outlines future work on capability and evaluation.

6. Conclusions and Future Work

  • The evaluation compares four configurations across four benchmarks, including GeoExpertQueries for expert workflows and GeoKTools for tool-type variation; development splits for both new benchmarks are released.The other benchmarks are CBGB and UnivEARTH.Source: §6 · pp. 24–25
  • GRA significantly outperforms standalone and basic-tool LLMs across GeoExpertQueries, GeoKTools, and CBGB; all examined systems score above 85 on UnivEARTH.On open-ended problems, the gap is clearest between full GRA with Gemini 3.8 Flash and the other configurations.Source: §6 · p. 25, §6.2 · p. 26
  • UnivEARTH appears saturated, while four CBGB reference values no longer matched the data; the authors argue geospatial benchmarks need periodic ground-truth revalidation.UnivEARTH standalone-LLM scores of 87 to 94 indicate answers are largely recoverable from the published source articles; CBGB drift followed imagery reprocessing and a boundary change.Source: §6 · p. 25
  • Because GeoExpertQueries scores alignment with expert workflows rather than outcome correctness, GRA outputs are best treated as first drafts for expert review.Its golden workflows specify steps rather than reference values.Source: §6 · p. 25

6.1. Geospatial Reasoning Capability

  • Current benchmarks do not exercise GRA’s spatiotemporal prediction tools and only lightly exercise Remote Sensing Foundations and WeatherNext.The authors plan targeted benchmarks for open-vocabulary visual search and ensemble weather forecasting.Source: §6.1 · p. 25
  • As planetary data continually change, the authors expect domain-specific tools and orchestration to remain important; modular skills let capabilities grow without retraining the model.The observed CBGB data drift illustrates the challenge of relying on fixed model parameters for changing planetary knowledge.Source: §6.1 · pp. 25–26

6.2. Evaluating Geospatial Reasoning Agents

  • The evaluation covers Gemini models only, excludes infeasible queries, and designs GeoKTools around operations supported by GRA’s tools, limiting the scope of its comparisons.The authors characterize GeoKTools as a test of multi-tool orchestration rather than general geospatial coverage; their GRA comparison also does not isolate individual orchestration components.Source: §6.2 · p. 26
  • Reliable rating of open-ended analyses remains an open problem, while enterprise-query evaluation faces data-availability, sensitivity, and reproducibility challenges.High-quality expert-workflow benchmarks are also expensive and time-consuming to produce.Source: §6.2 · p. 26

A.1. GRA Tool Suite

Table 11 catalogs GRA capabilities by pillar and cross-cutting function, alongside underlying data sources and models.

A.1. GRA Tool Suite

  • Table 11 catalogs GRA capabilities across five pillars and cross-cutting functions, with underlying data sources and models.Source: §A.1 · p. 31, Table 11 · p. 31
  • The table is presented as a capability inventory grouped by pillar.Source: Table 11 · p. 31
  • The entries continue across multiple table segments, as indicated by the continued captions.Source: Table 11 · pp. 32–33

A.2. Progressive Disclosure and Prefix Caching

Progressively disclosed skills can conserve context and support modular capabilities, but loading native tool declarations late can incur substantial serving costs.

A.2. Progressive Disclosure and Prefix Caching

  • Progressively disclosed skills trade off tool-calling reliability against serving efficiency.Source: §A.2 · p. 33
  • Native function declarations preserve function-calling alignment and schema-validated decoding, but changing tool configuration invalidates the cached prefix and forces conversation re-prefilling.Loading skills after many steps or large tool outputs have accumulated is correspondingly expensive.Source: §A.2 · p. 33
  • A generic runner tool avoids cache invalidation but forfeits dedicated function-calling training and schema validation, which can lower invocation success rates.Source: §A.2 · p. 33
  • GRA retains native declarations and mitigates cache costs by loading expected skills early and together, before large tool outputs accumulate.Source: §A.2 · p. 33

A.3. LLM-based Autoraters

Both autoraters use Gemini 3.8 Flash with medium thinking enabled by default.

A.3. LLM-based Autoraters

  • Both autoraters use Gemini 3.8 Flash with the default medium thinking level.Source: §A.3 · p. 34

A.3.1. GeoExpertQueries Workflow Autorater

The GeoExpertQueries autorater scores how much analytical value an agent’s executed workflow delivers relative to an expert consensus workflow. It treats that workflow as a reference, judging deviations by their practical impact rather than counting steps.

A.3.1. GeoExpertQueries Workflow Autorater

  • The autorater compares the query, consensus golden workflow, and execution trace, scoring delivered analytical value from 0 to 2.When experts provide alternative workflows, it evaluates the one the agent follows most closely.Source: §A.3.1 · pp. 34–35
  • The rubric evaluates executed data and workflow, not final arithmetic or conclusions, and ignores maps, charts, and formatted tables.A step described but not executed does not count; hardcoded empirical values do not substitute for tool-retrieved data.Source: §A.3.1 · p. 34
  • Deviations are weighed by their effect on usefulness to a geospatial expert, with real-data execution required for any nonzero score.Reasonable choices for unspecified parameters and an equally good fallback can be accepted; a fallback earns full credit only if an expert would regard it as equivalent.Source: §A.3.1 · pp. 34–35
  • A score of 0 applies to plan-only runs, missing final answers, no real analysis, or fatal deviations; a score of 2 allows expert-acceptable alternatives with at most slight loss.A score of 1 covers real analysis that addresses the main question but has significantly less usefulness due to deviations.Source: §A.3.1 · p. 36
  • Each item’s score averages three autorater runs after mapping scores 0, 1, and 2 to 0, 0.5, and 1.0.Results tables display the resulting 0–1 score on a 0–100 scale.Source: §A.3.1 · p. 36

A.3.2. GeoKTools Autorater

The GeoKTools autorater compares a response with a deterministic reference answer and assigns a continuous accuracy score. Its rubric calibrates errors to the query’s geospatial domain and applies explicit rules for special cases.

A.3.2. GeoKTools Autorater

  • The GeoKTools autorater compares the query, golden answer, and model response, assigning a continuous score from 0 to 1 for answer accuracy.The same autorater also scores CBGB answers with partial credit.Source: §A.3.2 · p. 36
  • Accuracy is judged by the gap in domain-relevant units, from equivalent answers to severe errors such as a wrong entity or missed threshold.Intermediate tiers distinguish negligible differences from errors that could mislead practitioners or materially change outcomes.Source: §A.3.2 · p. 36
  • The golden answer is authoritative, while rounding and formatting differences are tolerated and continuous errors receive proportional scores.For arrays, element scores are averaged; a wrong entity invalidates dependent metrics, and predicting events when the reference is zero is heavily penalized.Source: §A.3.2 · p. 37

B.1. Natural Hazards & Crisis Resilience: 2022 Pakistan Floods Damage & Reconstruction Prioritization

The example workflow estimates flood damage in Sindh and Balochistan by distinguishing inundation from building rubble in radar observations, then combining damage estimates with poverty and population data. The passage notes that SAR alone may not establish exterior structural damage reliably.

B.1. Natural Hazards & Crisis Resilience: 2022 Pakistan Floods Damage & Reconstruction Prioritization

  • The proposed analysis distinguishes standing water from rubble using pre- and post-flood SAR observations, then combines estimated damage with poverty and population data to prioritize reconstruction grants.The prompt focuses on hard-hit zones of Sindh and Balochistan after the 2022 floods.Source: §B.1 · pp. 37–38
  • Radar signal drops indicate standing water, while reduced coherence with rough, high radar return indicates possible building rubble.The analysis first aligns and cleans pre- and post-flood radar imagery and restricts the assessment to pre-existing towns.Source: §B.1 · p. 38
  • District-level prioritization combines destroyed-building density with poverty and population levels.The workflow aggregates the need assessment by union council.Source: §B.1 · p. 38
  • SAR alone may make exterior structural damage harder to establish, potentially requiring angled optical or ground-level imagery.Source: §B.1 · p. 38

B.2. Public Health: Emergency Obstetric Care Accessibility in Sierra Leone

The example estimates physical travel-time access to emergency obstetric care across Sierra Leone using a land-cover friction surface. It compares communities inside and outside a two-hour reach while noting that physical access does not establish effective service availability.

B.2. Public Health: Emergency Obstetric Care Accessibility in Sierra Leone

  • The analysis calculates travel time to qualifying maternity facilities from every location using a Dynamic World land-cover friction surface.The prompt frames this as physical accessibility rather than straight-line distance.Source: §B.2 · p. 38
  • The workflow assigns road and off-road travel speeds, slows movement on steep slopes, and blocks river crossings except at bridges or ferries.It computes minimum travel time from each 100m×100m grid cell to the nearest qualifying facility.Source: §B.2 · p. 38
  • The analysis applies a 120-minute benchmark and overlays population maps to count women of childbearing age inside and outside the reach in each district.Source: §B.2 · p. 38
  • The measure assumes vehicle availability and round-the-clock hospital staffing and supplies, so it captures physical rather than effective access.It does not assess areas at risk of being cut off during storms or floods.Source: §B.2 · p. 39

B.3. Site Selection & Urban Planning: Building Footprints vs. Available Land in Buenos Aires

The Buenos Aires example frames a city-wide audit of built-up, excluded, and buildable land. It outlines a workflow for estimating available land and identifies limits on what the estimate represents.

B.3. Site Selection & Urban Planning: Building Footprints vs. Available Land in Buenos Aires

  • The audit supports planning decisions about whether urban growth should extend onto vacant lots or grow upward amid pressure on infrastructure and green space.Source: §B.3 · p. 39
  • Building-layer preparation requires deciding what counts as a building and removing duplicate or overlapping footprints.Source: §B.3 · p. 39
  • The no-build mask combines roads, railways, rivers with safety buffers, and protected parks.Source: §B.3 · p. 39
  • Available buildable land is calculated by subtracting building footprints and no-build zones from total city land, then calculating percentages across neighborhoods.Source: §B.3 · p. 39
  • The city boundary matters: the Autonomous City is essentially built out, while metropolitan-area vacancy data are patchier and may require local-government cooperation.Source: §B.3 · p. 39
  • The measure captures physical vacancy, not legal buildability or vertical underuse, because zoning and building height are outside the 2D footprint analysis.Source: §B.3 · p. 39

Related papers

Loading 2610.15750v1…