PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents

Akashah Shabbir, Muhammad Umer Sheikh, Muhammad Akhtar Munir, Hiyam Debary, Mustansar Fiaz, Muhammad Zaigham Zaheer, Paolo Fraccaro, Fahad Shahbaz Khan, Muhammad Haris Khan, Xiao Xiang Zhu, Salman Khan

arXiv:2602.17665v4cs.CV

TL;DR

Geospatial agents must combine multimodal perception with spatially grounded, verifiable multi-step reasoning. OpenEarthAgent addresses this gap through a unified executable tool registry, validated reasoning trajectories, and a large multimodal corpus, achieving strong and balanced performance across geospatial reasoning and tool-use evaluations.

  • Problem

    Geospatial agents still face challenges in spatial grounding, coordinate consistency, and physically verifiable outputs during multi-step reasoning.

  • Method

    OpenEarthAgent combines multisensor EO data, GIS and index operations, explicit reasoning trajectories, and a unified executable tool registry with deterministic validation.

  • Results

    OpenEarthAgent achieves balanced gains across perception, operation, logic, and GIS evaluations while leading tool-order accuracy and outperforming general-agent baselines on a balanced benchmark subset.

  • Takeaways & Limitations

    The framework provides a unified setting for structured, interpretable, and verifiable geospatial reasoning across diverse EO tasks.

Abstract

from arXiv · show

Recent progress in multimodal reasoning has enabled agents that interpret imagery, connect it with language, and execute structured analytical tasks. Extending these capabilities to remote sensing remains challenging, as models must reason over spatial scale, geographic structures, and multispectral indices while maintaining coherent multi-step logic. To address this gap, we introduce \textit{OpenEarthAgent}, a unified framework for tool-augmented geospatial reasoning trained on satellite imagery, natural-language queries, and structured reasoning traces. Beyond serving as a benchmark, OpenEarthAgent establishes a cohesive agentic architecture built around a unified executable tool registry and trajectory-based policy learning. The framework standardizes heterogeneous visual, spectral, GIS, and georeferenced raster operations under a consistent callable schema, enabling modular orchestration and deterministic execution. Training is performed via supervised fine-tuning on structured reasoning trajectories with deterministic replay validation to ensure executability and spatial correctness. The accompanying corpus comprises 14,538 training and 1,169 evaluation instances with over 107K reasoning steps, spanning urban, environmental, disaster, and infrastructure domains and incorporating GIS operations alongside index analyses such as NDVI, NBR, and NDBI. Grounded in explicit reasoning traces, the learned agent demonstrates structured reasoning, stable spatial understanding, and interpretable tool-driven behaviour across diverse EO scenarios. We report consistent improvements over a strong baseline and competitive performance against recent open and closed-source models. Our code, data and trained models are publicly available: https://github.com/mbzuai-oryx/OpenEarthAgent

1 Introduction

OpenEarthAgent addresses limitations in geospatial grounding and verifiable multi-step reasoning by combining a unified executable tool registry with a trajectory-based training framework. Its corpus integrates multimodal EO data, GIS and index computations, and validated reasoning traces for broad geospatial evaluation.

  • Remote-sensing agents still struggle with geospatial grounding, coordinate consistency, and physically verifiable outputs.
  • OpenEarthAgent integrates GIS layers, multisensor imagery, and index analyses across urban, environmental, disaster, land-use, and transportation tasks.The framework includes operations such as distance, area, and zonal statistics alongside NDVI, NBR, and NDBI analyses.
  • The framework organizes tool calls, intermediate states, and outcomes into explicit trajectories for structured, interpretable workflows.
  • A unified tool registry standardizes perception, GIS, spectral, and georeferenced raster operations under an executable schema.The design couples the dataset with an execution engine and trajectory-based policy alignment.
  • Deterministic validation checks reasoning chains for spatial and logical consistency rather than merely generating tool-use traces.
  • 14,538 training instances and 1,169 evaluation tasks form a multimodal corpus for spatial reasoning, grounding, and interpretability.

2 Related Work

Prior EO systems increasingly combine multimodal perception with agentic tool use, but persistent gaps remain in geospatial grounding, coordinate consistency, and physically verifiable reasoning. OpenEarthAgent addresses this gap by unifying diverse geospatial tools, multimodal data, and supervised reasoning trajectories under an executable framework.

  • Related EO and agentic systems: EO research has progressed from foundation and vision-language models toward agentic systems that select and sequence external tools for multi-step reasoning.The cited works span remote-sensing foundation models, multimodal alignment, and tool-driven architectures.
  • Open challenges: Existing geospatial agents still face challenges in coordinate consistency, spatial grounding, and physically verifiable outputs.These limitations motivate agents that integrate perception with explicit, interpretable reasoning.
  • OpenEarthAgent: OpenEarthAgent unifies GIS, index-based, optical, and SAR operations through structured instruction-tool pairs and detailed reasoning traces.The framework also includes a dedicated evaluation set for assessing reasoning across diverse geospatial conditions.
  • Dataset curation: Its dataset construction combines multimodal sources, filtering, query synthesis, tool-grounded trajectories, and automated validation before training and evaluation.Filtering includes annotation checks, POI constraints, and spectral-change signals such as NDVI, NBR, and NDBI.
  • Methodology: The framework organizes perception and action into callable tool operations that an orchestrator validates, executes, and appends to the reasoning context.Tools use standardized JSON contracts with structured inputs, outputs, and executable functions.

4 Experiments

OpenEarthAgent is evaluated through step-by-step and end-to-end geospatial tool-use benchmarks against proprietary and open-source models. It shows strong action validity, balanced tool-selection performance, high trajectory fidelity, and leading results on several geospatial comparisons.

  • Baseline Evaluation: OpenEarthAgent achieves state-of-the-art Inst./Tool./ArgN./ArgV. accuracies in step-by-step evaluation despite its smaller parameter budget.Frontier models remain strongest on final summary accuracy.
  • End-to-End Evaluation: OpenEarthAgent delivers balanced Per./Op./Logic./GIS performance and dominates tool-order accuracy at approximately 67–72% in end-to-end evaluation.Its Answer Accuracy reaches 45.26, indicating strong trajectory planning beyond isolated tool correctness.
  • Cross-Benchmark Evaluation: OpenEarthAgent attains the highest Ans of 16.35 and Ans_I of 29.13 on the ThinkGeo comparison, surpassing ThinkGeo and the Qwen3-4B baseline.
  • General-Agent Comparison: OpenEarthAgent scores 60.04 overall against 21.75 for GPT-Agent and 22.90 for Qwen3-4B, while averaging 21.67 seconds per query.GPT-Agent averages 933.54 seconds and Qwen3-4B 42.02 seconds per query.
  • Category-Wise Performance: OpenEarthAgent achieves 79.43% on index-based tasks and 55.77% on GIS tasks, while open-source models perform comparatively better on image-based queries.Smaller models approach zero on index tasks and remain marginal on GIS and image-based tasks.

5 Conclusion

OpenEarthAgent connects natural language with multimodal earth-observation data through structured, executable analytical workflows. Its validated corpus supports grounded geospatial reasoning beyond perception-only modeling.

  • OpenEarthAgent unifies visual, spectral, GIS, and GeoTIFF-aware tools under a consistent executable schema for interpretable geospatial reasoning.The corpus contains 14,538 training instances and 1,169 held-out test instances with validated multi-step trajectories.
  • The framework moves EO modeling beyond perception by providing explicit tool calls, intermediate observations, and outputs for structured and verifiable reasoning.The paper positions this capability for environmental monitoring, disaster response, infrastructure analysis, and broader geospatial decision-making.

Supplementary

The supplementary material documents ablations, evaluation details, error analysis, and dataset-curation procedures supporting the main experiments.

  • The supplement includes ablations, model-selection criteria, error analysis, context-specific performance assessments, and quantitative evaluation cases.
  • It also documents input-output structures, tool descriptions, evaluation prompts, LLM-judge comparisons, and dataset-curation components.

S1 Ablation Analysis

Ablations show that data composition, in-context examples, tool-schema exposure, ordering, and explicit planning each affect geospatial tool reasoning. Mixed trajectories and fully specified interfaces provide the strongest overall behavior.

  • Data Composition: Easy-only training yields the strongest Perception F1 at 59.63, while hard-only training improves Operation F1 to 56.83 and tool-order robustness.Hard samples achieve AnyO 65.61 and Unique 70.40.
  • Data Composition: The combined setting achieves the highest Logic F1 of 51.18, GIS F1 of 98.52, Answer accuracy of 45.26, and General Accuracy of 75.86.It also produces the strongest tool-order consistency at approximately 67–73%.
  • In-Context Learning: Increasing in-context examples generally improves Qwen3 performance, but degradation in Operation and Generation between k=10 and k=15 indicates diminishing returns.OpenEarthAgent remains substantially above all Qwen variants across metrics.
  • Tool Schema Influence: Executable constraints raise Logical F1 by +6.72, while explicit argument signatures increase Perception by +5.23 and GIS by +11.61.These changes improve reasoning rigor and task–tool alignment without substantially changing tool-order metrics.
  • Tool Schema Influence: Full tool specifications produce the largest gains, including Operation +28.85, GIS +21.66, and Unique tool-order correctness +43.45.Explicit JSON examples provide maximal interface grounding by instantiating valid invocation patterns.
  • Tool Ordering: Optimized tool ordering improves Perception, Logic, and GIS F1, while Answer accuracy rises from 43.35 to 45.26.Ordering primarily improves execution reliability and in-distribution reasoning, with generation accuracy remaining stable.
  • High-Level Planning: Explicit high-level planning increases Logical reasoning F1 by +3.57, Any-order accuracy by +4.36, Same-order accuracy by +4.37, and image-generation accuracy by +10.34.

S2 Model Selection

Model selection compares two backbones under identical training and tool-conditioning settings, with explicit high-level planning improving structured execution. The smaller Qwen3-4B-Instruct-2507 performs slightly better than Qwen2.5-7B-Instruct across nearly all reported metrics.

  • Backbone comparison: Identical training, hyperparameter, and tool-conditioning settings are used to compare Qwen3-4B-Instruct-2507 with Qwen2.5-7B-Instruct.Training uses 25% of the corpus, with an internal 8% held-out validation split for model selection.
  • Planning strategy: Explicit high-level planning improves logical reasoning, tool sequencing consistency, image-generation decision accuracy, and overall structured execution.
  • Backbone comparison: Logic F1: +2.79 and AnyO: +7.0 are the strongest reported advantages for Qwen3-4B-Instruct-2507 over the 7B counterpart.The smaller model outperforms its counterpart across nearly all metrics, while both variants show near-saturated GIS and operational F1 scores.
  • Backbone comparison: Qwen3-4B-Instruct-2507 slightly outperforms Qwen2.5-7B-Instruct, indicating more efficient multi-step reasoning despite its smaller size.

S3 Error Analysis

Error analysis separates syntax violations from reasoning and tool-selection failures, showing that tool invocation remains a central failure mode. OpenEarthAgent achieves the lowest overall error count while maintaining low latency and balanced tool-use control.

  • Syntax errors: Syntax errors involve missing required actions or malformed JSON, with open-source models showing notably higher violation rates.Each conversation permits one thought-only planning turn; subsequent turns must include at least one valid action.
  • Reasoning errors: Answer-without-tool-call errors are rare, whereas multiple-tool-call errors are more prevalent in Mistral0.3-7B-Instruct and indicate over-generation.
  • Error taxonomy: Four error types are analyzed: No action, Wrong action format, Answer reached without a single tool call, and Multiple tool calls in a single action.
  • Comparative performance: OpenEarthAgent achieves the lowest total error count and the best balance between invoking necessary tools and avoiding omissions or redundancies.
  • Efficiency: GPT-5 and OpenEarthAgent achieve approximately 21.66 and 21.67 s/sample, respectively, while several open-source models require 98–137 s/sample.Tool-related failures, especially syntax errors and failures to call required tools, dominate the reported error patterns.

S4 Context-Specific Performance Assessment

The benchmark is stratified across geospatial domains to assess robustness under differences in scale, temporal dynamics, spectral ambiguity, and semantic complexity. OpenEarthAgent performs strongly in frequent domains and remains competitive in infrastructure and disaster tasks.

  • Evaluation scope: Domain-specific evaluation covers urban planning, environmental monitoring, recreation, industrial infrastructure, disaster assessment, aviation, transportation, and other categories.The stratification targets contextual robustness across varied geospatial task settings.
  • Comparative analysis: Inference time efficiency and error distributions vary across architectures, with tool-specific failure modes distinguished by model.
  • Domain challenges: Fine-grained recognition of small or sparse objects under varying resolutions and viewpoints demands stronger contextual and geometric reasoning in transportation tasks.
  • Results: OpenEarthAgent matches or surpasses GPT-4o and GPT-5 in urban planning, environmental monitoring, and recreation, while remaining competitive in infrastructure and disaster assessment.The reported pattern indicates effective generalization across operational geospatial reasoning scenarios.

S5 Qualitative Results

Qualitative examples show OpenEarthAgent composing executable multi-step workflows for spectral change mapping, metric-scale distance estimation, and buffered road-network analysis. These demonstrations connect natural-language requests to sequential tool use and interpretable geospatial outputs.

  • Environmental monitoring: A zero-shot environmental workflow extracts a park boundary, computes December 2024 and February 2025 NBR layers, derives their difference, and visualizes vegetation loss and recovery.The sequence uses GetAreaBoundary, AddIndexLayer, ComputeIndexChange, and ShowIndexLayer with coherent arguments.
  • Metric-scale reasoning: For metric-scale reasoning, the agent detects vehicles, computes their centroids, and calculates Euclidean distance using imagery with GSD 0.6.
  • Environmental monitoring: The NBR difference map shows pronounced negative values in southern Topanga, revealing burn scars and vegetation loss, while northern areas remain stable.
  • Spatial analysis: A buffered-area workflow adds points of interest, computes nearest-neighbor road distances from kindergartens to police stations, and visualizes the links on a base map.The example demonstrates sequential tool orchestration for spatial querying and interpretable visualization.
  • Workflow presentation: The qualitative examples present tool invocation alongside agent reasoning and visualization outputs for diverse geospatial tasks.

S6 Additional Tools Details

OpenEarthAgent combines low-level visual utilities, higher-level perception tools, GIS functions, and spectral operations in a broad executable toolkit.

  • The toolkit spans visual utilities, object and region analysis, change detection, and pixel-level segmentation.Examples include box drawing, text addition, OCR, object detection, region description, and segmentation.
  • GIS functions provide spatial operations such as boundary retrieval, POI layering, and distance computation.

S7 Evaluation Prompt

The evaluation framework scores geospatial answers using structured, modality-specific criteria and supports controlled comparison across judges. Despite Qwen3-30B’s higher absolute scores, model rankings remain consistent.

  • S7 Evaluation Prompt: Answer Accuracy produces a normalized score from 0 to 1 by comparing predicted answers with ground truth.
  • S7 Evaluation Prompt: Numerical answers use a ±10% tolerance threshold, while spatial outputs use Intersection over Union thresholds.
  • S7 Evaluation Prompt: The available-tools table lists each tool with its brief description, inputs, and outputs.
  • Open-Source LLM Judge: A Comparative Evaluation: Qwen3-30B replaces the proprietary judge while preserving the prompt, scoring criteria, and JSON-constrained format.
  • Open-Source LLM Judge: A Comparative Evaluation: Qwen3-30B assigns slightly higher absolute Answer Accuracy scores across models than the original judge.
  • Open-Source LLM Judge: A Comparative Evaluation: Model rankings remain preserved across judges, with stronger agents consistently outperforming weaker baselines.

S8 Detailed Description of Dataset-Curation Components

The curation pipeline transforms heterogeneous optical, SAR, GIS, and index-based sources into unified, executable samples. It filters task suitability, harmonizes annotations, preserves spatial inputs, and synthesizes tool-grounded queries.

  • The pipeline normalizes, filters, and validates heterogeneous earth-observation sources into spatially grounded, tool-compatible samples.
  • RGB-SAR ingestion extracts metadata, image paths, annotation files, and sensor-specific attributes from diverse datasets.
  • Task-dependent sufficiency filters require conditions such as visible objects, relevant entities, localized boxes, or paired event inputs.
  • The annotation harmonizer maps boxes, masks, pixel counts, categories, and labels into a common JSON-compatible schema.
  • The area candidate generator identifies geographic regions represented by names, boundaries, bounding boxes, or GeoPackage-compatible geometries.
  • Candidate regions support POI retrieval, distance computation, buffering, area estimation, and map visualization.
  • POI existence checks retain regions only when required categories and sufficient entities are available for the requested query.
  • The index branch mines NDVI, NBR, and NDBI events such as vegetation loss, burn damage, flooding, regrowth, and urban expansion.
Loading 2602.17665v4…