OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents

1 Introduction

• Existing remote-sensing vision-language models remain largely descriptive, while early tool-augmented EO agents still face geospatial-grounding, coordinate-consistency, and physically verifiable-output challenges.These limitations motivate agents that integrate perception with explicit, interpretable reasoning.Source: §1 · pp. 2–3
• The corpus links multisensor imagery, GIS layers, and index computations to explicit tool-call trajectories across urban, environmental, disaster, land-use, and transportation tasks.It includes GIS operations such as distance, area, and zonal statistics, and index analyses including NDVI, NBR, and NDBI.Source: §1 · p. 3
• The unified tool registry standardizes perception, GIS, spectral analysis, and georeferenced raster operations, while trajectory-based training enables deterministic validation of spatial and logical consistency.The executable schema supports consistent orchestration, tool syntax, and spatially grounded decision-making.Source: §1 · p. 3
• The benchmark contains 14,538 training instances and 1,169 evaluation tasks for studying spatial reasoning, grounding, and interpretability across reasoning and non-reasoning models.The held-out evaluation set is designed to assess model consistency and reasoning performance.Source: §1 · p. 3

2 Related Work

• Foundation and general tool-agent models offer scalable representations or modular tool use, but lack structured, verifiable geospatial reasoning, coordinate awareness, and scale handling.The related work describes early self-supervised pretraining, foundation models including Prithvi-v2 [38] and Galileo, and agent frameworks including ReAct, Voyager [42], and OctoTools.Source: §2 · p. 4
• EO agents address tool-augmented QA, spectral operations, or plan-tool-verify prompting, but retain weaknesses in spatial consistency, predefined workflows, or prompt-dependent policies; OpenEarthAgent unifies data, training, and evaluation.ThinkGeo [32], Earth-Agent, and Geo-OLM illustrate these limitations.Source: §2 · p. 4

3.1 Dataset Curation

• 14,538 training samples and 1,169 evaluation samples pair imagery and natural-language queries with structured reasoning traces across seven domains and optical, SAR, GIS, and multispectral data.The training and evaluation splits contain 100,656 and 7,064 reasoning steps, respectively; index layers include NDVI, NBR, and NDBI.Source: §3.1 · pp. 5, 8
• The corpus combines open-access imagery and annotations from benchmarks, Sentinel collections, and Google Earth Engine, spanning object-level to large-scale context and deriving index layers for environmental indicators.Its seven domains include urban planning, disaster assessment, environmental monitoring, transportation, aviation, recreation, and industrial infrastructure.Source: §3.1 · p. 5
• Curation harmonizes annotations, mines spectral changes, and serializes imagery, geospatial metadata, and tool arguments; synthesized queries and trajectories undergo tool-execution checks and manual test-set inspection.Validation checks argument correctness, geometric validity, and spatial consistency, with controlled regeneration or targeted manual verification for detected inconsistencies.Source: §3.1 · pp. 6–7, Figure 2 · p. 7

3.2 Methodology

• OpenEarthAgent turns multimodal instructions and inputs into iterative perception-action trajectories, using working memory for instructions, observations, spatial metadata, prior calls, and execution feedback.Supervised fine-tuning on validated traces targets syntactic validity, spatial consistency, and coherent multi-step reasoning.Source: §3.2 · p. 9
• A unified registry exposes visual, spectral, GIS, georeferenced-raster, and utility tools through standardized JSON contracts, while an orchestrator validates calls, executes them, and returns outputs to working memory.Cached intermediate layers and geometries prevent redundant computation and support deterministic replay; new tools can be registered without backbone retraining.Source: §3.2 · pp. 9–10
• Training optimizes tool-action predictions over verified trajectories, masks tool observations from the loss, and deterministically replays each trajectory to check formatting, coordinates, geometry, and executability.This keeps the execution environment separate from policy outputs while supporting spatially grounded, interpretable workflows.Source: §3.2 · pp. 10–11

4 Experiments

• OpenEarthAgent is fine-tuned from Qwen3-4B-Instruct-2507 for one epoch on four NVIDIA A100 GPUs, using a 2 × 10−5 learning rate and 4096-token maximum sequence length.Training uses response-only masking, computing loss on assistant-generated tokens rather than prompts or external tool outputs.Source: §4 · p. 11

4.1 Baseline Evaluation

• Evaluation uses tool-agnostic step-by-step rollouts to assess reasoning and tool invocation, plus end-to-end live-tool execution to measure operational robustness and task completion.Metrics include action validity, tool selection, argument-name and argument-value accuracy, summarization, category-wise tool F1, tool-order fidelity, and final task accuracy.Source: §4.1 · pp. 11–12
• OpenEarthAgent-4B achieves Inst./Tool./ArgN./ArgV. scores of 99.51/97.18/96.08/62.10, while o4-mini leads Summ. at 89.48.End-to-end, OpenEarthAgent has balanced tool-category gains, dominates tool-order accuracy at approximately 67–72%, and reaches Answer Accuracy 45.26; GPT-4o leads Op. F1 at 66.91 and Gen. Acc. at 77.93.Source: Table 3 · p. 12, §4.2 · p. 13, Table 4 · p. 13

4.2 Results

• On Earth-Agent, OpenEarthAgent-4B outperforms similarly sized open baselines on every tool-related metric and rivals GPT-4o; on ThinkGeo, it scores Ans 16.35 and Ans_I 29.13.Earth-Agent evaluation covers 248 tasks and 13,729 images; its tools are reformatted to the OpenEarthAgent interface with curated JSON exemplars.Source: §4.2 · pp. 13–14, Table 5 · p. 14, Table 6 · p. 14
• On the 60-query general-agent comparison, OpenEarthAgent scores 60.04 overall versus GPT-Agent’s 21.75 and Qwen3-4B’s 22.90, with 21.67-second mean latency.On spectrum tasks it scores 90.46 versus GPT-Agent’s 14.67; mean latency is 42.02 seconds for Qwen3-4B and 933.54 seconds for GPT-Agent.Source: §4.2 · p. 14, Table 6 · p. 14
• OpenEarthAgent executes more tool calls with a higher overall success rate, while Qwen2.5-7B, Qwen2.5-3B, and Llama-3.1-8B have 43–46% failure rates.Llama-3.1-8B and Mistral-7B fail to complete 94% and 98% of tasks, respectively; proprietary models have approximately 7% call error rates, and Mistral-7B under-invokes tools.Source: §4.2 · pp. 14–15, Figure 6 · p. 15
• Index scores are 79.43% for OpenEarthAgent, 79.39% for GPT-4o, and 78.94% for GPT-5; GIS scores are 55.77%, 54.94%, 41.95%, and 39.55%, respectively, for OpenEarthAgent, GPT-5, GPT-4o, and o4-mini.Smaller open models approach zero on index tasks and perform only marginally on GIS and image queries.Source: §4.2 · p. 15, Figure 7 · p. 15

• OpenEarthAgent orchestrates visual, spectral, GIS, and GeoTIFF-aware tools under a consistent executable schema for grounded, interpretable geospatial reasoning.Source: §5 · p. 15
• Its corpus contains 14,538 training instances and 1,169 held-out test instances with validated multi-step trajectories, including tool calls, intermediate observations, and outputs.Source: §5 · p. 15
• The framework moves EO research beyond perception-only modeling toward structured, verifiable reasoning for environmental monitoring, disaster response, infrastructure analysis, and geospatial decision-making.Source: §5 · p. 15

S1–S4

• Supplementary sections S1–S4 cover ablation analysis, model selection criteria, error analysis, and context-specific performance assessment.Source: Supplementary · p. 20

S5–S7

• Sections S5–S7 provide quantitative evaluation cases, input-output structures with tool descriptions, and an evaluation prompt with an LLM Judge comparison.Source: Supplementary · p. 20

S8

• Section S8 details the dataset-curation components used to construct the OpenEarthAgent corpus.Source: Supplementary · p. 20

Effect of Data Composition

• Combined training achieved the highest Logic F1 (51.18), GIS F1 (98.52), Answer (45.26), and General Accuracy (75.86), with the strongest tool-order consistency.Easy-only training yielded the highest Perception F1 (59.63), while hard-only training improved Operation F1 (56.83) and tool-order metrics (AnyO 65.61, Unique 70.40).Source: Effect of Data Composition · pp. 20–21, Table A1 · p. 20

Comparison with In-context Learning

• In-context examples improved Qwen3-Instruct-2507 over BL, but OpenEarthAgent outperformed every Qwen variant across all metrics; Operation and Generation degraded from k=10 to k=15.At k=10, Logical and Operation F1 and tool-order consistency improved; at k=15, Perception and Logical F1 and tool-order metrics improved.Source: Table A2 · p. 21, Comparison with In-context Learning · p. 22

Tool Schema Influence

• Semantic descriptions raised Perception (19.80→43.53) and Operation (8.40→27.24); signatures improved GIS (+11.61), while executable constraints raised Logic (+6.72).The schema variants progressively add semantic grounding, argument signatures, and executable requirements before maximal specification with JSON examples.Source: Table A3 · p. 22, Tool Schema Influence · pp. 23–24
• Full tool specification yielded the largest gains: Perception (+11.29), Operation (+28.85), Logical (+10.24), GIS (+21.66), and tool-order correctness.Explicit full JSON examples clarify argument schemas, formatting constraints, required fields, and executable structure, reducing ambiguity and malformed calls.Source: Table A3 · p. 22, Tool Schema Influence · p. 24

Effect of Tool Order

• Optimized tool ordering with explicit rules improved reasoning F1 across settings, including Perception (+6.80, +1.85), Logic (+3.93, +2.68), and GIS (+2.98, +1.06).Operation peaked at 59.54 with optimized ordering without rules; answer accuracy rose from 43.35 to 45.26 while generation accuracy remained stable.Source: Effect of Tool Order · p. 24, Table A4 · p. 25

Impact of High-Level Planning

• High-level planning increased Logical reasoning F1 by +3.57, Any-order accuracy by +4.36, Same-order accuracy by +4.37, and image generation accuracy by +10.34.Image generation accuracy increased from 65.52 to 75.86 when an explicit reasoning plan preceded tool execution.Source: Impact of High-Level Planning · p. 25

S2 Model Selection

• The comparison trained both backbones under identical data, hyperparameter, and tool-conditioning settings, using 25% of training data and an internal 8% held-out validation split for model selection.The validation split was separate from the benchmark test set.Source: S2 Model Selection · p. 25
• Both variants had near-saturated GIS and operational F1 scores, while answer and generation accuracy differed only marginally.The authors interpret this as evidence that reasoning quality is independent of model scale.Source: S2 Model Selection · p. 26
• Qwen3-4B-Instruct-2507 offers a stronger balance between model size and reasoning reliability for scalable geospatial applications.The authors attribute its advantage despite fewer parameters to architectural refinements and improved alignment in the Qwen3 series.Source: S2 Model Selection · p. 26, Table A6 · p. 26
• Logic F1 improved by +2.79 and AnyO by +7.0 for Qwen3-4B-Instruct-2507 over Qwen2.5-7B-Instruct, which outperformed across nearly all metrics.The comparison uses Table A6’s reasoning and tool-usage metrics.Source: Table A6 · p. 26, S2 Model Selection · p. 26

S3 Error Analysis

• OpenEarthAgent exhibits the fewest syntax errors; open-source models show more violations, while GPT-4o has substantially fewer syntax-related failures.Syntax failures include missing required actions and malformed JSON; only the initial planning turn may be thought-only.Source: S3 Error Analysis · p. 26
• Answer-without-tool-call errors are rare, except for Qwen3-4B-Instruct-2507 and o4-mini, while redundant calls are more prevalent in Mistral0.3-7B-Instruct.The analysis identifies missed tool calls and multiple calls in a single step as reasoning errors.Source: S3 Error Analysis · pp. 26–27
• OpenEarthAgent achieves the lowest total error count and best balance of tool use, avoiding both omitted and redundant calls.The error distribution covers four types: no action, wrong action format, reaching an answer without a tool call, and multiple tool calls in one action.Source: S3 Error Analysis · pp. 26–27, Figure A1 · p. 27
• Llama 3.1-8B, InternLM3-8B, and Mistralv0.3-7B take 98–137 s/sample, versus ∼21.66 for GPT-5 and ∼21.67 s/sample for OpenEarthAgent.Tool-related failures, primarily syntax errors and failures to call required tools, dominate the error patterns; GPT-4o, GPT-5, and OpenEarthAgent show substantially fewer errors across all categories.Source: S3 Error Analysis · p. 27

• The benchmark stratifies performance across urban planning, environmental monitoring, recreation, industrial infrastructure, disaster assessment, aviation, transportation, and other domains.This breakdown enables fine-grained evaluation of contextual robustness.Source: S4 Context-Specific Performance… · p. 27
• Domain difficulty reflects variation in spatial scale, temporal dynamics, spectral ambiguity, and semantic complexity; small or sparse-object tasks in industrial, aviation, and transportation settings demand stronger contextual and geometric reasoning.Urban planning and environmental monitoring are comparatively easier because they typically have stable large-scale patterns and consistent spectral signatures, while disaster assessment adds abrupt change, heterogeneous damage, and noise.Source: S4 Context-Specific Performance… · pp. 27–28
• OpenEarthAgent matches or surpasses GPT-4o and GPT-5 in urban planning, environmental monitoring, and recreation, while remaining competitive in infrastructure and disaster tasks.Open-source baselines show greater performance variance and lower accuracy on complex reasoning tasks.Source: S4 Context-Specific Performance… · p. 28

S5 Qualitative Results

• For a query about two vehicles at 0.6 GSD, the agent detects them, computes centroids, and calculates Euclidean distance using sequential tools.The example demonstrates metric-scale reasoning from imagery with precise cross-modal computation.Source: S5 Qualitative Results · pp. 28–29, Figure A5 · p. 30
• In a zero-shot environmental task, the agent extracts a park boundary, computes NBR layers for December 2024 and February 2025, and maps their difference to quantify vegetation loss and recovery.The trace shows coherent tool grounding and correctly formatted arguments across the workflow; the difference map shows burn scars in southern Topanga and stable northern areas.Source: S5 Qualitative Results · p. 28, Figure A4 · p. 29
• Within a 1000 m buffer, the agent adds POI layers, computes each kindergarten’s nearest-police-station road distance, and visualizes the POIs and distance links on a base map.This example integrates spatial queries and interpretable visualization in one reasoning pipeline.Source: S5 Qualitative Results · p. 30, Figure A6 · p. 30

Tools I/O Structure

• Defined input-output schemas let the agent compose perceptual, geometric, GIS, and spectral tools into transparent multi-step reasoning chains across imagery, metadata, and geospatial layers.The tools range from drawing, OCR, and segmentation to boundary retrieval, POI layers, distance computation, and NDVI or NDBI index layers.Source: S6 Additional Tools Details · p. 31

Tool Implementations

• The tool implementations pair TEOChat for change detection, LAE-DINO for object detection and localization, GEE and geemap for index and environmental analysis, OSMnx and NetworkX for spatial querying and network analysis, and Qwen3-VL-7B [2] for image understanding.These tools provide domain-specific capabilities for complex geospatial reasoning tasks.Source: S6 Additional Tools Details · p. 31
• Approximately 45 GB of runtime memory is required for the complete tool stack during evaluation, with timeout occurrence described as minimal.Source: S6 Additional Tools Details · p. 31

S7 Evaluation Prompt

• Answer Accuracy measures prediction alignment with ground truth on a normalized score from 0 to 1.The framework is illustrated in Fig. A8.Source: S7 Evaluation Prompt · p. 31
• Quantitative answers are scored using a ±10% tolerance threshold.Source: S7 Evaluation Prompt · p. 31
• Spatial answers are evaluated using Intersection over Union (IoU) thresholds.Source: S7 Evaluation Prompt · p. 31
• Descriptive and categorical answers are assessed for semantic equivalence, including paraphrases, partial correctness, and contradictions.Source: S7 Evaluation Prompt · p. 31
• Missing, fabricated, or non-responsive outputs receive zero where required elements are absent or invalid, and responses must use JSON with a score and concise justification.Source: S7 Evaluation Prompt · p. 31

Open-Source LLM Judge: A Comparative Evaluation

• Qwen3-30B assigns slightly higher absolute Answer Accuracy scores, but preserves the relative ranking of stronger agents over weaker baselines.The prompt, scoring criteria, and JSON-constrained output format were held identical for the cross-judge comparison.Source: Open-Source LLM Judge: A Comparative… · pp. 32–33, Figure A7 · p. 33

RGB-SAR Data Ingestion

• RGB-SAR ingestion indexes source data, filters samples against task-specific annotation requirements, and harmonizes retained annotations into a shared schema for perception tools.Insufficient objects, ambiguous labels, missing boxes, invalid masks, or incomplete paired inputs are discarded; labels and fields are normalized, with deterministic derivations and added annotations where needed.Source: RGB-SAR Data Ingestion · pp. 34–35, Sample Sufficiency Filter · p. 35, Annotation Harmonizer · p. 35

Area Candidate Generator

• The GIS branch selects candidate regions, exposes compatible point-of-interest layers, and rejects areas lacking sufficient valid POIs for the requested spatial queries.Regions may include buffers and are chosen for operations such as distance, buffering, area estimation, and accessibility; POI categories come from sources such as OpenStreetMap.Source: Area Candidate Generator · pp. 35–36, Available POI Layers · p. 36, POI Existence Filter · p. 36

Index/Event Miner

• The index branch mines candidate events using NDVI, NBR, and NDBI, then retains spatially coherent changes that pass predefined thresholds.Target patterns include vegetation loss, burn damage, regrowth, flooding, or urban expansion; weak or noisy differences are excluded.Source: Index/Event Miner · p. 36, Index Change Gate · p. 37

Unified Data Representation

• Unified records combine modality-specific metadata and executable input references so the question synthesizer and tool orchestrator can use a consistent interface.Records retain image or raster references, annotations, spatial metadata, tool arguments, and task categories; image and geospatial inputs include file paths, sensor or layer details, coordinate systems, and raster metadata.Source: Unified Data Representation · p. 37, Image and GeoPackage Information · p. 37

Question Prompt Suite

• Task-specific prompt templates vary query wording while preserving tool dependencies, and the synthesizer generates grounded questions with expected input references and candidate tool arguments.Queries with mismatched objects, invalid regions, missing dates, or unsupported tool requirements are rejected or regenerated.Source: Question Prompt Suite · pp. 37–38, Question Synthesizer · p. 38

Conversational One-Shot Generator

• The one-shot generator demonstrates category-specific thought-action-observation-answer trajectories, while the dataset prompt suite constrains complete examples to valid tools, arguments, and input references.Examples encode different tool sequences for RGB-SAR, GIS, and index queries, supporting consistent formatting and executable tool-policy learning.Source: Conversational One-Shot Generator · p. 38, Dataset Prompt Suite · p. 38

Dataset Constructor

• The dataset constructor combines each query, multimodal inputs, metadata, and reasoning trace into a standardized training or evaluation instance.Entries preserve tool calls, observations, final answers, task and modality types, tool sequence, and reasoning length, enabling supervision over the full process.Source: Dataset Constructor · p. 39

Validation Suite

• The validation suite admits instances only after checking tool-call syntax, replaying trajectories, validating spatial outputs, and manually inspecting held-out evaluation samples.Failures are discarded or regenerated; evaluation inspection targets unrealistic measurements, ambiguous queries, and invalid metadata.Source: Validation Suite · p. 39